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Preface 



There is a growing body of research and develop- 
ment work in the emerging field of neuroergonom- 
ics. For the first time, this book brings together this 
body of knowledge in a single volume. In compos- 
ing this book, we sought to show how an under- 
standing of brain function can inform the design of 
work that is safe, efficient, and pleasant. Neuroer- 
gonomics: The Brain at Work shows how neuroer- 
gonomics builds upon modern neuroscience and 
human factors psychology and engineering to en- 
hance our understanding of brain function and 
behavior in the complex tasks of everyday life, as- 
sessed outside the confines of the standard research 
laboratory, in natural and naturalistic settings. 

The book begins with an overview of key is- 
sues in neuroergonomics and ends with a view 
toward the future of this new interdisciplinary 
field. Specific topics are covered in 22 intervening 
chapters. The subject matter is wide ranging and 
addresses scientific and clinical approaches to diffi- 
cult questions about brain and behavior that con- 
tinue to drive our investigations and the search for 
solutions. This composition required the input of 
specialists with a variety of insights on medicine, 
human factors engineering, physiology, psychol- 
ogy, neuroimaging, public health policy, and the 
law. Effective response to these issues requires 



coordinated efforts of many relevant specialists, 
utilizing shared knowledge and cross-fertilization 
of ideas. We hope this book contributes to these 
ends. 

The breadth and depth of this volume would 
not have been possible without the steady influ- 
ence and vision of Series Editor Alex Kirlik and the 
Oxford University Press. We are also extremely 
indebted to the authors for their creative contribu- 
tions and timely responses to our extensive edito- 
rial advice. Raja Parasuraman was supported by 
grants from the National Institutes of Health and 
DARPA and Matthew Rizzo by the National Insti- 
tutes of Health and the Centers for Disease Control 
and Prevention. Raja Parasuraman is grateful to 
former members of the Cognitive Science Labora- 
tory, especially Francesco DiNocera, Yang Jiang, 
Bernd Lorenz, Ulla Metzger, and Sangy Panicker, 
for stimulating debates in the early days of neu- 
roergonomics, many carried out online and to con- 
tinuing discussions with current members including 
Daniel Caggiano, Shimin Fu, Pamela Greenwood, 
Reshma Kumar, Ericka Rovira, Peter Squire, and 
Maria Zinni, and to the other members of the Arch 
Lab at George Mason University. Matt Rizzo thanks 
his colleagues in neurology, engineering, public 
health, and the Public Policy Center for their 
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open-minded collaboration and is especially obliged Evelyn for nurturing his curiosity to Annie, Elite, 

to the past and present members of the Divi- and Frannie for their enduring support, and to Big 

sion of Neuroergonomics (http://www.uiowa.edu/- Bill and Margie, now at peace. Both of us are also 

neuroerg/) for their good humor, great ideas, and grateful to Constance Kadala and to Cheryl Moores 

hard work. He is deeply grateful to Michael and for their editorial assistance. 
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Raja Parasuraman and Matthew Rizzo 



Introduction to Neuroergonomics 



Neuroergonomics is the study of brain and behav- 
ior at work (Parasuraman, 2003). This interdisci- 
plinary area of research and practice merges the 
disciplines of neuroscience and ergonomics (or 
human factors) in order to maximize the benefits 
of each. The goal is not just to study brain struc- 
ture and function, which is the province of neuro- 
science, but also to do so in the context of human 
cognition and behavior at work, at home, in trans- 
portation, and in other everyday environments. 
Neuroergonomics focuses on investigations of the 
neural bases of such perceptual and cognitive 
functions as seeing, hearing, attending, remember- 
ing, deciding, and planning in relation to tech- 
nologies and settings in the real world. Because 
the human brain interacts with the world via a 
physical body, neuroergonomics is also concerned 
with the neural basis of physical performance — 
grasping, moving, or lifting objects and one's 
limbs. 

Whenever a new interdisciplinary venture 
is proposed, it is legitimate to ask whether it is 
necessary. To answer this query, we show how 
the chapters in this book, as well as other work, 
demonstrate that neuroergonomics provides added 
value, beyond that available from "traditional" 
neuroscience and "conventional" ergonomics, to 



our understanding of brain function and behavior 
as it occurs in the real world. The guiding princi- 
ple of neuroergonomics is that examining how the 
brain carries out the complex tasks of everyday 
life — and not just the simple, artificial tasks of 
the research laboratory — can provide important 
benefits for both ergonomics research and prac- 
tice. An understanding of brain function can lead 
to the development and refinement of theory in 
ergonomics, which in turn will promote new, far- 
reaching types of research. For example, knowl- 
edge of how the brain processes visual, auditory, 
and tactile information can provide important 
guidelines and constraints for theories of infor- 
mation presentation and task design. The basic 
premise is that the neuroergonomic approach al- 
lows the researcher to ask different questions and 
develop new explanatory frameworks about hu- 
mans and work than an approach based solely 
on the measurement of the overt performance or 
subjective perceptions of the human operator. The 
added value that neuroergonomics provides is 
likely to be even greater for work settings such as 
modern semiautomated systems (Parasuraman & 
Riley, 1997), where measures of overt user behav- 
ior can be difficult to obtain (Kramer & Weber, 
2000). 
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Some Examples of 
Neuroergonomics Research 

Aviation 

The following examples illustrate the value of the 
neuroergonomic approach. Historically, the great- 
est influence of human factors on technological 
design has been in the domain of aviation, specif- 
ically in the design of displays and controls in the 
aircraft cockpit (Fitts, Jones, & Milton, 1950; 
Wiener & Nagel, 1988). With the worldwide 
growth in airline travel, new proposals for air traf- 
fic management have been put forward. Imple- 
menting these proposals requires new cockpit 
technologies. Consider a new traffic-monitoring 
system that is to be installed in the cockpit to por- 
tray to the pilot other aircraft that are in the im- 
mediate vicinity, showing their speed, altitude, 
flight path, and so on, using color-coded symbols 
on a computer display. Various types of neuroer- 
gonomic research, both basic and applied, can in- 
form the design of this system. For example, 
designers may wish to know what features of the 
symbols (e.g., shape, intensity, motion, etc.) serve 
to best attract the pilot's attention to a potential 
intruder in the immediate airspace. At the same 
time, there may be a concern that the presenta- 
tion of traffic information, while helping the pilot 
monitor the immediate airspace, may increase 
the pilot's overall mental workload, thereby de- 
grading the performance of the primary flight 
task. Although subjective or performance mea- 
sures could be used to evaluate this possibility, a 
neuroergonomic approach can provide more sen- 
sitive evaluation of any impact on flight perfor- 
mance. It may also lead the researcher to ask new 
and potentially more profitable questions about 
attention allocation than before. Measures of 
brain function that reflect visual attention and 
oculomotor control can help determine the im- 
pact of the new display on the pilot's visual scan- 
ning and attentional performance (see chapter 7, 
this volume). Finally, neuroergonomic evaluation 
of the manual and physical demands involved in 
interacting with the information panels and con- 
trols of the new traffic-monitoring system would 
also be required for this system to be used effec- 
tively and safely by pilots (see chapter 15, this 
volume). 



Driving 

Neuroergonomics is also relevant to assessing 
interactions between the eye, the brain, and the 
automobile (Rizzo & Kellison, 2004). Functional 
magnetic resonance imaging (fMRI) permits nonin- 
vasive dynamic imaging of the human brain (see 
chapter 4, this volume). Analytic approaches to 
fMRI data, such as independent component analy- 
sis, can reveal meaningful patterns in data sets col- 
lected in subjects performing complex tasks that 
capture elements of automobile driving. Prelimi- 
nary application of these approaches suggests that 
multiple neural regions, including the frontopari- 
etal, cerebellar, and occipital areas, are differen- 
tially activated by various aspects of the driving 
task, such as speed control. It is also possible to 
relate physiological correlates of impending sleep 
(microsleeps) derived from electroencephalo- 
graphic (EEG) activity recordings of brain activity 
to imminent declines in driver performance (Paul, 
Boyle, Rizzo, & Tippin, 2005). Finally, naturalistic 
studies of driver behavior provide unique evidence 
of long-range human interactions, strategies, and 
tactics of "the brain in the wild" (see chapter 8, this 
volume). 

Neuroenginering 

A third example concerns the use of brain signals 
as an additional communication channel for hu- 
man interaction with both the natural and the 
human-made environment. This area of research 
and practice, sometimes also called neuroengineer- 
ing or brain-computer interface (BCI), has had signif- 
icant progress in recent years. In this approach, 
different types of brain signals are used to control 
external devices without the need for motor out- 
put, which would be advantageous for individuals 
who either have only limited motor control or, 
as in the case of "locked-in" patients with amy- 
otrophic lateral sclerosis, virtually no motor con- 
trol. The idea follows naturally from the work on 
"biocybemetics" in the 1980s pioneered by Donchin 
and others (Donchin, 1980; Gomer, 1981) but has 
progressed beyond the earlier achievements with 
technical developments in recording of brain activ- 
ity in real time. 

BCls allow a user to interact with the environ- 
ment without engaging in any muscular activity, for 



Introduction to Neuroergonomics 



example, without the need for hand, eye, foot, or 
mouth movement. Instead, the user is trained to en- 
gage in a specific type of mental activity that is asso- 
ciated with a unique brain electrical "signature." The 
resulting brain potentials are recorded, processed, 
and classified in such a way as to provide a control 
signal in real time for an external device. Applica- 
tions have used a variety of different measures of 
brain electrical activity. Invasive methods include 
recording of field potentials and multiunit neuronal 
activity from implanted electrodes; this technique 
has been reported to be successful in controlling ro- 
botic arms (Nicolelis, 2003). Such invasive record- 
ing techniques have superior signal-to-noise ratio 
but are obviously limited in use to animals or to 
patients with no motor functions in whom elec- 
trode implantation is clinically justified. Noninva- 
sive BCIs have used a variety of brain signals derived 
from scalp EEG recordings. These include quan- 
tified EEGs from different frequency bands such 
as beta and mu waves (Pfurtscheller & Neuper, 
2000), event-related potentials (ERPs) such as P300 
(Donchin, Spence, & Wijesinghe, 2000), and con- 
tingent negative variation (Birbaumer et al., 1999). 
BCIs based on these signals have been used to oper- 
ate voice synthesizers, control cursor movements on 
a computer display, and move robotic arms. 

Virtual Reality 

Virtual reality (VR) is particularly relevant to neu- 
roergonomics because VR can replicate situations 
with greater control than is possible in the real 
world, allowing behavioral and neural measures of 
the mind and brain at work in situations that are 
impractical or impossible to observe in the real 
world. In doing so, VR can be used to study the per- 
formance of human operators engaged in hazardous 
tasks without putting them and others at risk for in- 
jury (see chapter 17, this volume). For example, VR 
can be used to study the influence of disease, drugs, 
fatigue, or in-vehicle technologies (such as cell 
phones) on aircraft piloting and automobile driv- 
ing, to study how to reduce the risk of falls in the 
elderly, and to train students to avoid novice mis- 
judgments and errors in performing critical med- 
ical procedures, flying aircraft, and operating heavy 
machinery. VR is particularly useful in workers 
whose jobs require spatial awareness, complex mo- 
tor skills, or decisions that require evaluation of 



multiple possible responses amid changing contin- 
gencies, and is also proving to be useful for therapy 
and rehabilitation of persons with motor, cognitive, 
and psychiatric impairments. 



Conceptual, Theoretical, 
and Philosophical Issues 

The constituent disciplines of neuroergonomics — 
neuroscience and ergonomics/human factors 
research — are both twentieth-century post-World 
War II fields. The spectacular rise of neuroscience 
toward the latter half of that century and the 
smaller but no less important growth in human fac- 
tors research can both be linked to technological 
developments in computer science and engineer- 
ing. The brain imaging technologies that have revo- 
lutionized modern neuroscience (e.g., fMRI) and 
the sophisticated automated systems that have stim- 
ulated much human factors work (e.g., the aircraft 
flight management system) were both made possi- 
ble by these engineering developments. Neverthe- 
less, the two fields have developed independently. 

Traditionally, ergonomics has not paid much 
attention to neuroscience or to the results of 
studies concerning brain mechanisms underlying 
human perceptual, cognitive, affective, and motor 
processes. At the same time, neuroscience and its 
more recent offshoot, cognitive neuroscience, has 
only been recently been concerned with whether 
its findings bear any relation to human functioning 
in real (as opposed to laboratory) settings. Recent 
calls to move neuroscience "beyond the bench" 
("Taking Neuroscience beyond the Bench," 2002) 
include studies of group social behavior (Cacciopo, 
2002) and the development of neural prosthetics 
for control of robots, home automation, and other 
technologies for physically disabled people (see 
chapter 19, this volume). 

The relative neglect by ergonomics of human 
brain function is understandable given that this 
discipline had its roots in a psychology of the 
1940s that was firmly in the behaviorist camp. 
More recently, the rise of cognitive psychology in 
the 1960s influenced human factors, but for the 
most part neuroscience continued to be ignored by 
cognitive theorists, a state of affairs consistent with 
a functionalist approach to the philosophy of mind 
(Dennett, 1991). Such an approach implies that 



Introduction 



the characteristics of neural structure and function- 
ing are largely irrelevant to the development of the- 
ories of mental functioning. Cognitive psychology 
(and cognitive science) also went through a func- 
tionalist period in the 1970s and 1980s, mainly 
due to the influence of researchers from artificial 
intelligence and computer science. However, the 
recent influence of cognitive neuroscience has led 
to a retreat from this position. Cognitive neuro- 
science proposes that neural structure and function 
constrain and in some cases determine theories of 
human mental processes (Gazzaniga, 2000). 

If neuroscience has freed cognitive science 
from rigid functionalism, then ergonomics may 
serve to liberate it from a disembodied existence 
devoid of context and provide it an anchor in the 
real world. Even though researchers are aware of 
the importance of ecological validity, modern cog- 
nitive psychology (with a few exceptions) tends to 
study mental processes in isolation, apart from the 
artifacts and technologies of the world that require 
the use of those processes. Technology, particularly 
computers, can be viewed as an extension of hu- 
man cognitive capability. Related to this view is the 
framework of cognitive engineering, in which hu- 
mans and intelligent computer systems constitute 
"joint cognitive systems" (Hutchins, 1995; Roth, 
Bennett, & Woods, 1987). Furthermore, much hu- 
man behavior is situated and context dependent. 
Context is often defined by and even driven by 
technological change. How humans design, in- 
teract with, and use technology — the essence of 
ergonomics — should therefore also be central to 
cognitive science. 

The idea that cognition should be considered 
in relation to action in the world has many an- 
tecedents. Piaget's (1952) work on cognitive devel- 
opment in the infant and its dependence on 
exploration of the environment anticipated the 
concept of situated or embodied cognition. Clark 
(1997) also examined the characteristics of an em- 
bodied mind that is shaped by and helps shape ac- 
tion in a physical world. If cognitive science should 
therefore study the mind not in isolation but in in- 
teraction with the physical world, then it is a natu- 
ral second step to ask how to design artifacts in the 
world that best facilitate that interaction. This is the 
domain of ergonomics or human factors. Neuroer- 
gonomics goes one critical step further. It postulates 
that the human brain, which implements cognition 
and is itself shaped by the physical environment, 



must also be examined in interaction with the envi- 
ronment in order to understand fully the interrela- 
tionships of cognition, action, and the world of 
artifacts. 

Currently, a coherent body of concepts and 
empirical evidence that constitutes neuroergonom- 
ics theory does not exist. Of course, broad theories 
in the human sciences are also sparse, whether in 
ergonomics (Hancock & Chignell, 1995) or in 
neuroscience (Albright, Jessell, Kandel, & Posner, 
2001). Sarter & Sarter (2003) proposed that neu- 
roergonomics must follow the same reductionist 
approach of cognitive neuroscience in order to de- 
velop viable theories. There are small-scale theories 
that could be integrated into a macrotheory, but 
which would still pertain only to a specific domain 
of human functioning. For example, neural theo- 
ries of attention are becoming increasingly well 
specified, both at the macroscopic level of large- 
scale neural networks (Parasuraman, 1998; Posner, 
2004) and at the level of neuronal function and 
gene expression (Parasuraman, Greenwood, Ku- 
mar, & Fossella, 2005; Sarter, Givens, & Bruno, 
2001). At the same time, psychological theories of 
attention have informed human factors research 
and design (Wickens & Hollands, 2000). Difficult 
though the task may be, one can envisage amalga- 
mation of these respective theories into a neuroer- 
gonomic theory of attention. Integration across a 
broader range of functional domains, however, is 
as yet premature. 



Methods 

A number of methods have been developed for use 
in neuroergonomic research and practice. Among 
these are brain imaging techniques, which have 
been influential in the development of the field 
of cognitive neuroscience. Brain imaging tech- 
niques can be roughly divided into two classes. 
The first group of techniques is based on measure- 
ment of cerebral hemodynamics (blood flow), such 
as positron emission tomography (PET), fMRI, and 
transcranial Doppler sonography (TCD). The sec- 
ond group of methods involves measurement of 
the electromagnetic activity of the brain, including 
EEG, ERPs, and magnetoencephalography (MEG). 
For a review of brain imaging techniques for use in 
studies of cognition and human performance, see 
Cabeza and Kingstone (2001). 
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Figure 1.1. Resolution space of brain imaging techniques for ergonomic applications. 
Trade-offs between the criteria of the spatial resolution (y-axis) and temporal resolution 
(x-axis) of neuroimaging methods in measuring neuronal activity, as well as the relative 
noninvasiveness and ease of use of these methods in ergonomic applications (color code). 
EEG = electronencephalography; ERPs = event-related potentials; fMRI = functional 
magnetic resonance imaging; MEG = magnetoencephalography; NIRS = near-infrared 
spectroscopy; PET = positron emission tomography; TCDS = transcranial doppler 
sonography. See also color insert. 



PET and fMRI currently provide the best nonin- 
vasive imaging techniques for the evaluation and lo- 
calization of neural activity. However, these methods 
suffer from two drawbacks. First, their temporal res- 
olution is poor compared to electrophysiological 
techniques such as ERPs. Second, their use is re- 
stricted to highly controlled lab environments in 
which participants must not move. This limits their 
use for examining the neural basis of performance in 
more complex tasks with a view to ergonomic appli- 
cations, as in flight, driving simulation, or the use of 
virtual reality systems, although components of com- 
plex task performance are being studied (Peres, Van 
de Moortele, & Pierard, 2000; Calhoun et al„ 2002; 
see also chapter 4, this volume). Optical imaging 
techniques such as fast near-infrared spectroscopy 
(NIRS) may provide both spatial and temporal reso- 
lution and the ability to be used in neuroergonomic 
applications (see chapter 5, this volume). 

An overview of the relative merits and disad- 
vantages of these various techniques is shown in 
figure 1.1. This illustration is a variant of a repre- 



sentation of the spatiotemporal resolution of brain 
imaging methods first described by Churchland and 
Sejnowski (1988). The ease of ergonomic applica- 
tion (color code) has been added to the trade-off be- 
tween the criteria of spatial resolution and temporal 
resolution in measuring neuronal activity. Currently, 
there is no one technique that reaches the ideal (blue 
circle) of 0.1 mm spatial resolution, 1 ms temporal 
resolution, and ease of use in ergonomics. 

In addition to brain imaging methods, oculo- 
motor techniques can provide additional tools for 
neuroergonomics researchers. With the advent of 
low-cost, high-speed systems for measuring differ- 
ent types of eye movements and increasing knowl- 
edge of the underlying neural systems, oculomotor 
measures can provide important information not 
available from traditional measurement of response 
accuracy and speed (see chapter 7, this volume). 

It should be noted that the use of brain imag- 
ing or oculomotor measures need not be a defining 
characteristic of neuroergonomic research. A neu- 
roergonomic study may use behavioral measures or 
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a computational analysis; however, in each case the 
performance measure or the computational model 
is related to a theory of brain function. 

Consider the following example. Suppose that 
as a result of the manipulation of some factor, 
performance on a target discrimination task (e.g., 
detection of an intruder aircraft in the cockpit 
traffic-monitoring example discussed previously) in 
which location cues are provided prior to the target 
yields the following results: reaction time (RT) to 
the target when preceded by an invalid location 
cue is disproportionately increased, while that to a 
valid cue is not. This might happen, for example, 
if the cue is derived from the output of an auto- 
mated detection system that is not perfectly reliable 
(Hitchcock et al., 2003). In simple laboratory tasks 
using such a cueing procedure, there is good evi- 
dence linking this performance pattern to a basic 
attentional operation and to activation of a specific 
distributed network of cortical and subcortical 
regions on the basis of previous research using 
noninvasive brain imaging in humans, invasive 
recordings in animals, and performance data from 
individuals who have suffered damage to these 
brain regions (Posner & Dehaene, 1994). One 
could then conduct a study using the same cueing 
procedure and performance measures as a behav- 
ioral assay of the activation of the neural network 
in relation to performance of a more complex task 
in which the same basic cognitive operation is 
used. If the characteristic performance pattern was 
observed — a disproportionate increase in RT fol- 
lowing an invalid location cue, with a normal de- 
crease in RT following a valid cue — then one could 
argue that the distributed cortical/subcortical net- 
work of brain regions is likely to have been in- 
volved in task performance. This would then 
enable the researcher to link the full body of neuro- 
science work on this aspect of attentional function 
to performance on the complex intruder-detection 
task. Thus, even though no physiological index was 
used, and although the same performance measure 
(RT) was used as in a traditional ergonomic analy- 
sis, the type of question asked and the explanatory 
framework can be quite different in the neuroer- 
gonomic approach. 

Finally, a neuroergonomic study could also 
involve a computational analysis of brain or cog- 
nitive function underlying performance of a com- 
plex task. So long as the analysis was theoretically 
driven and linked to brain function, the study 



would qualify as neuroergonomic even though no 
physiological index was used. Several computational 
models of human performance have been developed 
for use in human factors (Pew & Mavor, 1998). Of 
these, models that can be linked, in principle, to 
brain function — such as neural network (connec- 
tionist) models (O'Reilly & Munakata, 2000) — 
would be of relevance to neuroergonomics. 



Neuroergonomics and 
Neuropsychology 

Neuropsychology and related fields (e.g., behav- 
ioral neurology, clinical and health psychology, 
neuropsychiatry, and neurorehabilitation) have also 
helped pave the way for neuroergonomics. Hebb 
(1949) used the term neuropsychology in his classic 
book The Organization of Behavior: A Neuropsycho- 
logical Theory. The field broadly aims to understand 
how brain structure and function are related to 
specific psychological processes. The neuropsycho- 
logical approach uses statistical techniques for 
standardizing psychological tests and scales to 
provide clinical diagnostic and assessment tools in 
normal and impaired individuals (de Oliveira Souza, 
Moll, & Eslinger, 2004). 

Like neuroergonomics, neuropsychology is 
dedicated to a psychometric approach, holding 
that human behavior can be quantified with ob- 
jective tests of verbal and nonverbal behavior, 
including neural states, and that these data reflect 
a person's states of mind and information pro- 
cessing. These processes can be divided into dif- 
ferent domains, such as perception, attention, 
memory, language, executive functions (decision 
making and implementation), and motor abilities, 
and they can be assessed using a wide variety of 
techniques. 

Both neuropsychology and neuroergonomics 
rely on principles of reliability (how repeatable a 
behavioral measure is) and validity (what a mea- 
sure really shows about human brain and behav- 
ior). Neuropsychology has traditionally relied on 
paper-and-pencil tests, many of which are stan- 
dardized and well understood (e.g., Lezak, 1995). 
The neuroergonomics approach is more rooted in 
technology, as indicated in this book. Novel tech- 
niques and tests are developing at a rapid pace, 
and guidelines and standards are going to be 
needed. 
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Contributions to Neuroergonomics 
from Other Fields: 
Genetics, Biotechnology, 
and Nanotechnology 

While we have emphasized the contribution of neu- 
roscience to neuroergonomics in this chapter, devel- 
opments in other fields are also affecting the study of 
human brain function at work. Three such fields are 
molecular genetics, biotechnology, and nanotech- 
nology, and we briefly consider their relevance here. 

As discussed previously, cognitive psychology 
has increasingly capitalized on findings from neu- 
roscience. More recently, the study of individual 
differences in cognitive function is being influ- 
enced by developments in molecular genetics and, 
in particular, the impressive results of the Human 
Genome Project. Much of what we know about 
the genetics of cognition has come from twin 
studies in which identical and fraternal twins are 
compared to assess the heritability of a trait. This 
paradigm has been widely used in behavioral ge- 
netics research for over a century. For example, 
the method has been used to show that general in- 
telligence, or g, is highly heritable (Plomin & 
Crabbe, 2000). However, this approach cannot 
identify the particular genes involved in intelli- 
gence or the cognitive components of g. Recent 
advances in molecular genetics now allow a differ- 
ent, complementary approach to behavioral gene- 
tics, called allelic association. This method has 
been applied to the study of individual differences 
in cognition in healthy individuals, revealing evi- 
dence of modulation of cognitive task perfor- 
mance by specific neurotransmitter genes (Fan, 
Fossella, Sommer, Wu, & Posner, 2003; Green- 
wood, Sunderland, Friz, & Parasuraman, 2000; 
Parasuraman et al., 2005). This work is likely to 
provide the basis not only for improved under- 
standing of the neural basis of cognition, but also 
for better characterization of individual differ- 
ences in cognition. That, in turn, can have an im- 
pact on important human factors issues such as 
selection and training. 

Reliable quantification of individual differ- 
ences in cognitive function will have obvious 
implications for selection of operators for occupa- 
tions that demand a high workload. While it 
would be premature to state that the molecular ge- 
netic approach to cognition has immediate appli- 
cations to selection, further programmatic research 



on more complex cognitive tasks will undoubtedly 
lead to progress in such an endeavor. The postge- 
nomic era has clearly demonstrated that inheri- 
tance of a particular genotype only sets a range for 
the phenotypic expression of that genotype, with 
the exact point within that range being determined 
by other genetic and environmental factors. Ge- 
nomic analysis allows for a much more precise 
specification of that range for any phenotype, and 
for linking phenotypic variation to specific genetic 
polymorphisms. Selection and training have tradi- 
tionally been considered together in human factors 
research and practice (e.g., Sanders & McCormick, 
1983) but rarely in terms of a common biological 
framework. Examining the effects of normal gene- 
tic variation and of various training regimens on 
brain function may provide such a common frame- 
work. 

The goal of neuroergonomics is to better 
understand the brain's functional structures and 
activities in relation to work and technology. In 
addition to molecular genetics, biotechnology can 
contribute to this effort by providing a means to 
study neuronal activities down to the molecular 
level. Biomimetic studies also allow for precise mod- 
eling of the human brain's activities. If the validity 
of such models can be established in the near fu- 
ture, then researchers could examine various ma- 
nipulations of brain function that are not ethically 
possible with human participants. 

The currently available measures of brain 
function are limited by sensor size and the inabil- 
ity to monitor brain function and influence func- 
tion simultaneously. Nanotechnology provides the 
measurement tools that can achieve such dual- 
purpose needs. It can also provide new sensors for 
monitoring changes in neuronal function in other- 
wise undetectable brain structures. In addition, 
nanotechnology has the appropriate scale of oper- 
ations necessary to deliver chemicals needed to 
precisely monitor and modify effects of neuro- 
transmitters or encourage targeted neurogenesis, 
with the objective of improving human performance 
in certain work environments. 

Although there are few current examples of 
the influence of biotechnology and nanotechnol- 
ogy on neuroergonomics, these fields are likely to 
have greater impact in the near future. De Pont- 
briand (2005) provided a cogent discussion of the 
potential benefits that biotechnology and nan- 
otechnology can bring to neuroergonomics. 
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Overview of Neuroergonomics: 
The Brain at Work 

This book represents a collective examination of 
the major theoretical, empirical, and practical is- 
sues raised by neuroergonomics. In this opening 
chapter, which forms part I, we have provided an 
overview of the field, covering theoretical and con- 
ceptual issues involved in the merging of cognitive 
neuroscience and human factors research. We have 
also briefly described neuroergonomic methods, 
but these are covered in more detail in part II, 
which consists of seven chapters describing differ- 
ent cognitive neuroscience methods: fMRI, EEG, 
ERPs, NIRS, TCD, and oculomotor measures. In 
addition, measures to track behavior and brain 
function in naturalistic environments are also de- 
scribed. Each chapter outlines the major features of 
each method, describes its principal merits and 
limitations, and gives illustrative examples of its 
use to address issues in neuroergonomics. We un- 
derstand that readers will bring a variety of techni- 
cal backgrounds to the examination of these 
methodological issues. Accordingly, key readings 
provided at the end of each chapter provide addi- 
tional background for understanding some of the 
more technical details of each method, as needed. 

Part III examines basic research in a number of 
different domains of cognition that have particular 
relevance for the understanding of human perfor- 
mance at work. We did not attempt to be com- 
prehensive. Rather, we chose areas of cognition in 
which significant progress has been made in identi- 
fying the underlying neural mechanisms, thereby 
allowing for theory-driven application to human 
factors issues. The cognitive domains discussed are 
spatial cognition, vigilance, executive functions, and 
emotion and decision making. In addition, work- 
ing memory, planning, and prospective memory 
are variously described in some of these chapters as 
well as in other sections of the book. 

As the study of the brain at work, neuroer- 
gonomics must also examine the work environment. 
It is an undeniable fact that many work settings are 
stressful, induce fatigue, and are poorly designed in 
terms of workspace layout. Accordingly, part IV ex- 
amines issues of stress, sleep loss, and fatigue, as 
well as the effects of the physical work environment. 

Part V consists of four chapters that discuss 
several different domains of application of neuroer- 
gonomics. Again, we did not attempt to cover all of 



the application areas that are emerging as a result 
of the use of neuroergonomic research. We chose 
four: adaptive automation, virtual reality, robotics, 
and neuroengineering. 

Neuroengineering applications are designed in 
part to help individuals with different disabilities 
that make it difficult for them to communicate 
effectively with the world. This area of work is 
covered in more detail in part VI. Four chapters 
describe neuroergonomic technologies that can be 
used to help the paralyzed, individuals with low or 
no vision, and those who require prostheses. A final 
chapter in this section is concerned with the evalua- 
tion of medical safety in health care settings. 

Finally, in part VII, we close the volume by sur- 
veying prospects for the future of neuroergonomics. 



Conclusion 

Neuroergonomics represents a deliberate merger of 
neuroscience and ergonomics with the goal of ad- 
vancing understanding of brain function underly- 
ing human performance of complex, real-world 
tasks. A second major goal is to use existing and 
emerging knowledge of human performance and 
brain function to design technologies and work en- 
vironments for safer and more efficient operation. 
More progress has been made on the first goal than 
on the second, but both neuroergonomic research 
and practice should flourish in the future, as the 
value of the approach is appreciated. The basic en- 
terprise of ergonomics — how humans design, inter- 
act with and use technology — can be considerably 
enriched if we also consider the human brain that 
makes such activities possible. 



Main Points 

1 . Neuroergonomics is the study of brain and 
behavior at work. 

2. Neuroergonomics attempts to go beyond its 
constituent disciplines of neuroscience and 
ergonomics by examining brain function and 
cognitive processes not in isolation but in 
relation to the technologies and artifacts of 
everyday life. 

3. Some examples of neuroergonomics include 
research in the areas of aviation, driving, 
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brain-computer interfaces, and virtual 
reality. 

4. Neuroergonomics is inconsistent with a 
purely functional philosophy of mind, 
in which brain structure and function 
are deemed irrelevant. In addition, 
neuroergonomics views brain and mind as 
influenced by context and technology. 

5. Neuroergonomic methods include behavioral 
and performance studies, brain imaging, 
oculomotor measures, and computational 
techniques. These methods have different 
relative merits and disadvantages. 
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Electroencephalography (EEG) 
in Neuroergonomics 



This chapter considers the utility of the ongoing, 
scalp-recorded, human electroencephalogram (EEG) 
as a tool in neuroergonomics research and practice. 
The EEG has been extensively documented to be a 
sensitive index of changes in neuronal activity due 
to variations in the amount or type of mental activity 
an individual engages in, or to changes in his or her 
overall state of alertness and arousal. The EEG is 
recorded as a time-varying difference in voltage be- 
tween an active electrode attached to the scalp and a 
reference electrode placed elsewhere on the scalp or 
body. In the healthy waking brain, the peak-to-peak 
amplitude of this scalp-recorded signal is usually 
well under 100 microvolts, and most of the signal 
power comes from rhythmic oscillations below a 
frequency of about 30 Hz. In many situations, the 
EEG is recorded simultaneously from multiple elec- 
trodes at different positions on the scalp, often 
placed over frontal, parietal, occipital, and temporal 
lobes of the brain according to a conventional place- 
ment scheme. 

The scalp-recorded EEG signal reflects postsy- 
naptic (dendritic) potentials rather than action (ax- 
onal) potentials. Since the laminar structure of the 
cerebral cortex facilitates a large degree of electrical 
summation (rather than mutual cancellation) of 
these postsynaptic potentials, the extracellular EEG 



recorded from a distance represents the passive 
conduction of currents produced by summating 
synchronous activity over large neuronal popula- 
tions. Several factors determine the degree to which 
potentials arising in the cortex will be recordable at 
the scalp, including the amplitude of the signal at 
the cortex, the size of a region over which postsy- 
naptic potentials are occurring in a synchronous 
fashion, the proportion of cells in that region that 
are in synchrony, the location and orientation of 
the activated cortical regions in relation to the scalp 
surface, and the amount of signal attenuation and 
spatial smearing produced by conduction through 
the highly resistive skull and other intervening 
tissue layers. While most of the scalp-recordable 
signal in the ongoing EEG presumably originates 
in cortical regions near the recording electrode, 
large signals originating at more distal cortical loca- 
tions can also make a significant contribution to 
the activity observed at a given scalp recording site. 
For example, because of the orientation of the pri- 
mary auditory cortices, some EEG signals gener- 
ated in them project more toward the top of the 
head than to the geometrically closer lateral scalp 
surfaces. 

The decomposition of an instantaneous scalp- 
recorded voltage measure into the constituent set 
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of neuronal events throughout the brain that con- 
tributed to it is a mathematically ill-conditioned 
inverse problem that has no unique solution. 
Because of this indeterminacy, the EEG has signifi- 
cant limitations with respect to its use as a method 
for three-dimensional anatomical localization of 
neural activity in the same sense in which func- 
tional magnetic resonance imagining (fMRI) or 
positron emission tomography (PET) are used. 
However, the EEG has obvious advantages relative 
to other functional neuroimaging techniques as a 
method for continuous monitoring of brain func- 
tion, either over long periods of time or in environ- 
ments such as a hospital bed. Indeed, it is often the 
method of choice for some clinical monitoring 
tasks. For example, continuous EEG monitoring is 
an essential tool in the diagnostic evaluation of 
epilepsy and in the evaluation and treatment of 
sleep disorders. It is also coming to play an increas- 
ingly important role in neurointensive care unit 
monitoring and in gauging level of awareness dur- 
ing anesthesia. 

For many years, efforts have also been un- 
der way to evaluate the extent to which the EEG 
might be useful as a monitoring modality in the 
context of human factors research. To be most 
useful in such settings, a monitoring method 
should be robust enough to be reliably measured 
under relatively unstructured task conditions, sen- 
sitive enough to consistently vary with some di- 
mension of interest, unobtrusive enough to not 
interfere with operator performance, and inexpen- 
sive enough to eventually be deployable outside 
of specialized laboratory environments. It should 
also have reasonably good time resolution to allow 
tracking of changes in mental status as complex be- 
haviors unfold. The EEG appears to meet such re- 
quirements. Furthermore, the compactness of EEG 
technology also means that, unlike other func- 
tional neuroimaging modalities (which typically 
require large expensive measuring instruments 
and complete immobilization of the subject), EEGs 
can even be collected from an ambulatory sub- 
ject wearing a lightweight and nonencumbering 
headset. 

A monitoring capability with such characteris- 
tics could provide unique value in the context of 
neuroergonomics research that seeks to better un- 
derstand the neurobiological impact of task condi- 
tions that impose excessive cognitive workload or 
that result in significant mental fatigue. The need 



for expansion of knowledge in this area is evi- 
denced by the extensive literature indicating that 
task conditions that impose cognitive overload of- 
ten lead to performance errors even in alert indi- 
viduals working under routine conditions. The 
potential for compromised performance in such 
circumstances can be exacerbated in individuals 
who are debilitated because of fatigue or sleep loss, 
illness or medication, or intoxication or hangover. 
In fact, even modest amounts of sleep loss can de- 
grade performance on tests that require contribu- 
tions from prefrontal cortical regions that control 
attention functions (Harrison & Home, 1998, 
1999; Harrison, Home, & Rothwell, 2000; Linde 
& Bergstrom, 1992; Smith, McEvoy, & Gevins, 
2002; see also chapter 14, this volume) and the 
magnitude of the behavioral impairment observed 
on such tasks can exceed that observed following a 
legally intoxicating dose of alcohol (Arendt, Wilde, 
Munt, & MacLean, 2001; Krull, Smith, Kalbfleisch, 
& Parsons, 1992; Williamson & Feyer, 2000). 

While most often just a barrier to productivity, 
some critical jobs are particularly demanding in 
terms of the fatigue and cognitive workload they 
impose, and are particularly unforgiving in terms 
of the severe negative consequences that can be in- 
curred when individuals performing those jobs 
make mistakes. For instance, in medical triage and 
crowded emergency room contexts the patient's life 
often hinges on a physicians ability to manage 
complex, competing demands, often after long 
hours on the job (Chisholm, Collison, Nelson, & 
Cordell, 2000). Similarly, the sleep deprivation and 
circadian desynchronization imposed by shift work 
scheduling has been noted to be a source of severe 
performance decrements (Scott, 1994) and has 
been implicated as a probable cause in a number of 
aviation (Price & Holley, 1990) and locomotive 
(Tepas, 1994) accidents. The high personal and so- 
cietal costs associated with such performance fail- 
ures motivate efforts to develop advanced methods 
for detecting states of cognitive overload or mental 
fatigue. 

In this chapter, we review progress in develop- 
ing EEG methods for such purposes. We first de- 
scribe how the spectral composition of the EEG 
changes in response to variations in task difficulty 
or level of alertness during highly controlled cogni- 
tive tasks. We also consider methods for analysis 
of such signals that might be suitable for use in a 
continuous monitoring context. Finally, we review 
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generalizations of those methods to assess com- 
plex, computer-based tasks that are more represen- 
tative of real-world tasks. 



EEG Signals Sensitive to Variations 
in Task Difficulty and Mental Effort 

A significant body of literature exists concerning the 
EEG changes that accompany increases in cognitive 
workload and the allocation of mental effort. One 
approach to this topic has focused on EEG changes 
in response to varying working memory (WM) de- 
mands. WM can be construed as an outcome of 
the ability to control attention and sustain its focus 
on a particular active mental representation (or set 
of representations) in the face of distracting influ- 
ences (Engle, Tuholski, & Kane, 1999). In many 
ways, this notion is nearly synonymous with what 
we commonly understand as effortful concentration 
on task performance. WM plays an important role 
in comprehension, reasoning, planning, and learn- 
ing (Baddeley, 1992). Indeed, the effortful use of 
active mental representations to guide performance 
appears critical to behavioral flexibility (Goldman- 
Rakic, 1987, 1988), and measures of it tend to be 
positively correlated with performance on psycho- 
metric tests of cognitive ability and other indices of 
scholastic aptitude (Carpenter, Just, & Shell, 1990; 
Gevins & Smith, 2000; Kyllonen & Christal, 1990). 
Many EEG studies of WM have required 
subjects to perform controlled n-back-style tasks 
(Gevins & Cutillo, 1993; Gevins et al, 1990, 1996) 
that demand sustained attention to a train of stim- 
uli. In these tasks, the load imposed on WM varies 
while perceptual and motor demands are kept rela- 
tively constant. For example, in a spatial variant of 
the n-back task, stimuli are presented at different 
spatial positions on a computer monitor once every 
4 or 5 seconds while the subject maintains a cen- 
tral fixation. Subjects must compare the spatial lo- 
cation of each stimulus with that of a previous 
stimulus, indicating whether a match criterion is 
met by making a key press response on a com- 
puter mouse or other device. In an easy, low-load 
version of the task, subjects compare each stimu- 
lus to the first stimulus presented in each block of 
trials (0-back task). In more difficult, higher-load 
versions, subjects compare the position of the cur- 
rent stimulus with that presented one, two, or 
even three trials previously (1-, 2-, or 3-back tasks). 



These require constant updating of the information 
stored in WM on each trial, as well as constant 
attention to new stimuli and maintenance of pre- 
viously presented information. To be successful 
in such tasks when WM demands are high, sub- 
jects typically must exert significant and continu- 
ous mental effort. Similar n-back tasks have been 
used to activate WM networks in a controlled fash- 
ion in the context of functional neuroimaging stud- 
ies employing PET or fMRI methods (Braver et al., 
1997; Cohen et al, 1994; Jansma, Ramsey, Cop- 
pola, & Kahn, 2000). 

The spectral composition of the ongoing EEG 
displays regular patterns of load-related modula- 
tion during n-back task performance. For example, 
figure 2.1 displays spectral power in the 4-14 Hz 
range at a frontal midline (Fz) and a parietal midline 
(Pz) scalp location computed from the continuous 
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Figure 2.1 . Effect of varying the difficulty of an n-back 
working memory task on the spectral power of EEG 
signals. The figure illustrates spectral power in dB of 
the EEG in the 4-14 Hz range at frontal (Fz) and pari- 
etal (Pz) midline electrodes, averaged over all trials of 
the tasks and collapsed over 80 subjects. Data from 
Gevins and Smith (2000). 
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EEG during performance of low-load (0-back) and 
moderately high-load (2 -back) versions of a spatial 
n-back task. The data represent the average re- 
sponse from a group of 80 subjects in a study of 
individual differences in cognitive ability (Gevins 
& Smith, 2000) and show significant differences 
in spectral power as a function of task load that 
vary between electrode locations and frequency 
bands. 

More specifically, at the Fz site a 5-7 Hz or 
theta-band spectral peak is increased in power dur- 
ing the high-load task relative to the low-load task. 
This type of frontal midline theta signal has fre- 
quently been reported to be enhanced in difficult, 
attention-demanding tasks, particularly those re- 
quiring a sustained focus of concentration (Gevins 
et al., 1979; Gevins et al., 1998; Gevins, Smith, 
McEvoy, & Yu, 1997; Miyata, Tanaka, & Hono, 
1990; Mizuki, Tanaka, Iogaki, Nishijima, & Inanaga, 
1980; Yamamoto & Matsuoka, 1990). Topographic 
analyses have indicated that this task loading- 
related theta signal tends to have a sharply defined 
potential field with a focus in the anterior midline 
region of the scalp (Gevins et al., 1997; Inouye 
et al., 1994); such a restricted topography is un- 
likely to result from distributed generators in dor- 
solateral cortical regions. Instead, attempts to 
model the generating source of the frontal theta 
rhythm from both EEG (Gevins et al., 1997) and 
magnetoencephalographic (Ishii et al, 1999) data 
have implicated the anterior cingulate cortex as a 
likely region of origin. This cortical region is thought 
to be part of an anterior brain network that is criti- 
cal to attention control mechanisms and that is acti- 
vated by the performance of complex cognitive tasks 
(Posner & Rothbart, 1992). Indeed, in a review of 
over 100 PET activation studies that examined ante- 
rior cingulate cortex activity, Paus and colleagues 
found that the major source of variance that af- 
fected activation in this region was associated with 
changes in task difficulty (Paus, Koski, Caramanos, 
& Westbury 1998). The EEG results are thus con- 
sistent with these views, implying that perfor- 
mance of tasks that require significant mental effort 
places high demands on frontal brain circuits in- 
volved with attention control. 

Figure 2.1 also indicates that signals in the 
8-12 Hz or alpha band tend to be attenuated in the 
high-load task relative to the low-load task. This in- 
verse relationship between task difficulty and alpha 
power has been observed in many studies in which 



task difficulty has been systematically manipulated. 
In fact, this task correlate of the alpha rhythm has 
been recognized for over 70 years (Berger, 1929). 
Because of this load-related attenuation, the magni- 
tude of alpha activity during cognitive tasks has 
been hypothesized to be inversely proportional to 
the fraction of cortical neurons recruited into a tran- 
sient functional network for purposes of task perfor- 
mance (Gevins & Schaffer, 1980; Mulholland, 1995; 
Pfurtscheller & Klimesch, 1992). This hypothesis is 
consistent with current understanding of the neural 
mechanisms underlying generation of the alpha 
rhythm (reviewed in Smith, Gevins, Brown, Karnik, 
& Du, 2001). Convergent evidence for this view is 
also provided by observations of a negative correla- 
tion between alpha power and regional brain activa- 
tion as measured with hemodynamic measures 
(Goldman, Stern, Engel, & Cohen, 2002; Moos- 
mann et al., 2003) and the frequent finding from 
neuroimaging studies of greater and more extensive 
brain activation during task performance when task 
difficulty increases (Bunge, Klingberg, Jacobsen, & 
Gabrieli, 2000; Carpenter, Just, & Reichle, 2000). 

In addition to signals in the theta and alpha 
bands, other spectral components of the EEG have 
also been reported to be sensitive to changes in ef- 
fortful attention. These include slow-wave activity 
in the delta (<3 Hz) band (McCallum, Cooper, & 
Pocock, 1988), high-frequency activity in the beta 
(15-30 Hz) and gamma (30-50 Hz) band (Sheer, 
1989), and rarely studied phenomena such as the 
kappa rhythm that occurs around 8 Hz in a small 
percentage of subjects (Chapman, Armington, & 
Bragden, 1962). 



Automated Detection of Mental 
Effort or Fatigue-Related 
Changes in the EEG 

The results reviewed above indicate that spectral 
components of the EEG vary in a predictable fashion 
in response to variations in the cognitive demands of 
tasks. While this is a necessary condition for the de- 
velopment of EEG-based methods for monitoring 
cognitive workload, a number of other issues must 
also be addressed if such laboratory observations are 
to be transitioned into practical tools. Foremost 
among them is the problem of EEG artifact. That 
is, in addition to brain activity, signals recorded at 
the scalp include contaminating potentials from eye 
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movements and blinks, muscle activity, head move- 
ments, and other physiological and instrumental 
sources of artifact. Such contaminants can easily 
mask cognition- related EEG signals (Barlow, 1986 
Gevins, Doyle, Schaffer, Callaway, & Yeager, 1980 
Gevins, Zeitlin, Doyle, Schaffer, & Callaway, 1979 
Gevins, Zeitlin, Doyle, Yingling, et al., 1979; Gevins, 
Zeitlin, Yingling, et al., 1979), an essential but diffi- 
cult and often subtle issue that, unfortunately, is too 
often given lip service but not actually dealt with. In 
laboratory studies, human experts review the raw 
data, identify artifacts and eliminate any contami- 
nated EEG segments to ensure that data used in 
analyses represent actual brain activity. For large 
amounts of data, this is an expensive, labor-intensive 
process which itself is both subjective and variable. 
To be practical in more routine applied contexts, 
such decisions must be made algorithmically. 

We have directed a great deal of research toward 
automated artifact detection. This has led to the de- 
velopment and testing of multicriteria spectral de- 
tectors (Gevins et al., 1975; Gevins, Yeager, Zeitlin, 
Ancoli, & Dedon, 1977), sharp transient waveform 
detectors (Gevins et al, 1976), and detectors using 
neural networks (Gevins & Morgan, 1986, 1988). 
In some cases, automated detection algorithms can 
perform about as well as the consensus of expert hu- 
man judges. For example, in a database of about 
40,000 eye movement, head/body movement, and 
muscle artifacts, we found that algorithmic methods 
successfully detected 98.3% of the artifacts with a 
false detection rate of 2.9%, whereas on average ex- 
pert human judges found 96.5% of the artifacts with 
a 1.7% false detection rate. Thus, while further work 
on the topic is needed, it is reasonable to expect that 
the problem of automated artifact detection will not 
be an insurmountable barrier to the development of 
EEG-based cognitive monitoring methods. 

A closely related problem is the fact that, in sub- 
jects actively performing tasks with significant per- 
ceptuomotor demands in a normal fashion, the 
incidence of data segments contaminated by arti- 
facts can be high. As a result, it can be difficult to 
obtain enough artifact-free data segments for analy- 
sis. To minimize data loss, effective digital signal 
processing methods must also be developed to filter 
contaminants out of the EEG when possible. One 
powerful approach to this problem has been to 
implement adaptive filtering methods to decon- 
taminate artifacts from EEG signals (Du, Leong, & 
Gevins, 1994). We have found such methods to 



be effective at recovering most of the artifact- 
contaminated data recorded in typical laboratory 
studies of subjects working on computer-based 
tasks. A variety of other methods have been em- 
ployed by different investigators in response to this 
problem, including such techniques as autoregres- 
sive modeling (Van den Berg-Lensssen, Brunia, & 
Blom, 1989), source-modeling approaches (Berg & 
Scherg, 1994), and independent components analy- 
sis (Jung et al, 2000). A difficult issue with contam- 
inant removal is that bona fide brain signals can also 
be removed with the artifacts. As with the problem 
of artifact detection, continued progress in this area 
suggests that, at least under some conditions and for 
some types of artifacts, decontamination strategies 
will evolve that will enable the automation of EEG 
processing for continuous monitoring applications. 

Presuming then that automated preprocessing 
of the EEG can yield sufficient data for subsequent 
analyses, questions still remain as to whether the 
type of load-related changes in EEG signals can be 
measured in a reliable fashion in individual sub- 
jects, and whether such measurements can be ac- 
complished with a temporal granularity suitable for 
tracking complex behaviors. That is, in the experi- 
ments described above, changes in the theta and al- 
pha bands in response to variations in WM load 
were demonstrated by collapsing over many min- 
utes of data recorded from a subject at each load 
level, and then comparing the mean differences be- 
tween load levels across groups of subjects using 
conventional parametric statistical tests. Under 
normal waking conditions, such task-related EEG 
measures have high test-retest reliability when com- 
pared across a group of subjects measured during 
two sessions with a week between them (McEvoy, 
Smith, & Gevins, 2000). However, for the devel- 
opment of automated EEG analysis techniques 
suitable for monitoring applications, load-related 
changes in the EEG would ideally also be replicable 
when computed over short segments of data and 
would need to have high enough signal-to-noise ra- 
tios to be measurable within such segments. 

Prior work has demonstrated that multivariate 
combinations of EEG variables can be used to ac- 
curately discriminate between specific cognitive 
states (Gevins, Zeitlin, Doyle, Schaffer, et al., 1979; 
Gevins, Zeitlin, Doyle, Yingling, et al., 1979; Gevins, 
Zeitlin, Yingling, et al., 1979; Wilson & Fisher, 
1995). Furthermore, neural network-based pat- 
tern classification algorithms trained on data from 
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individual subjects can also be used to automati- 
cally discriminate data recorded during different 
load levels of versions of the type of n-back WM 
task described above. For example, in one experi- 
ment (Gevins et al., 1998) eight subjects per- 
formed both spatial and verbal versions of 3-, 2-, 
and 1-back WM tasks on test sessions conducted 
on different days. For each single trial of data in 
each subject, spectral power estimates were com- 
puted in the theta and alpha bands for each elec- 
trode site. Pattern recognition was performed with 
the classic Joseph-Viglione neural network algo- 
rithm (Gevins, 1980; Gevins & Morgan, 1988; 
Joseph, 1961; Viglione, 1970). This algorithm iter- 
atively generates and evaluates two-layered feed- 
forward neural networks from the set of signal 
features, automatically identifying small subsets of 
features that produce the best classification of ex- 
amples from the sample of data set aside for train- 
ing. The resulting classifier networks were then 
cross-validated on the remaining data not included 
in the training sample. 

Utilizing these procedures, test data segments 
from 3-back versus 1-back load levels could be dis- 
criminated with over 95% (p < .001) accuracy. 
Over 80% (p < .05) of test data segments associ- 
ated with a 2-back load could also be discrimi- 
nated from data segments in the 3-back or 1-back 
task loads. Such results provide initial evidence 
that, at least for these types of tasks, it is possible to 
develop algorithms capable of discriminating dif- 
ferent cognitive workload levels with a high degree 
of accuracy. Not surprisingly, they also indicated 
that relatively large differences in cognitive work- 
load are easier to detect than smaller differences, 
and that there is an inherent trade-off between the 
accuracy of classifier performance and the tempo- 
ral length of the data segments being classified. 

High levels of accurate classification were also 
achieved when applying networks trained with 
data from one day to data from another day and 
when applying networks trained with data from 
one task (e.g., spatial WM) to data from another 
task (e.g., verbal WM). It was also possible to suc- 
cessfully apply networks trained with data from a 
group of subjects to data from new subjects. Such 
generic networks were found on average to yield 
statistically significant classification results when 
discriminating the 1-back from the 3-back task 
load conditions, but their accuracy was much re- 
duced from that achievable with subject-specific 



networks. On the one hand, such results indicate 
that there is a fair amount of commonality across 
days, tasks, and subjects in the particular set of 
EEG frequency-band measures that are sensitive 
to increases in cognitive workload. Such common- 
alities can be exploited in efforts to design efficient 
sensor montages and signal-processing methods. 
Nonetheless, they also indicate that to achieve 
optimal performance using EEG-based cognitive 
load-monitoring methods, it will likely be neces- 
sary to calibrate algorithms to accommodate in- 
dividual differences. Such conclusions are also 
consistent with the observation that patterns of 
task-related EEG changes vary in conjunction with 
individual differences in cognitive ability and cog- 
nitive style (Gevins & Smith, 2000). 

In addition to being sensitive to variations in 
attention and mental effort, the EEG also changes 
in a predictable fashion as individuals become 
sleepy and fatigued, or when they experience 
other forms of transient cognitive impairment. For 
example, it has long been known that the EEG of 
drowsy subjects has diffusely increased lower theta 
band activity and decreased alpha band activity 
(Davis, Davis, Loomis, Harvey, & Hobart, 1937; 
Gevins, Zeitlin, Ancoli, & Yeager, 1977). These 
changes are distinct from those described above 
characterizing increasing task load based on topog- 
raphy and spectral characteristics. Because such 
EEG changes are robust and reliable, a number of 
laboratories have developed and tested computer- 
ized algorithms for automated detection of drowsi- 
ness (Gevins, Zeitlin, et al, 1977; Hasan, Hirkoven, 
Varri, Hakkinen, & Loula, 1993). Such methods 
have produced highly promising results. For exam- 
ple, in one study we used neural network-based 
methods to compare task-related EEG features be- 
tween alert and drowsy states in individual sub- 
jects performing the n-back WM tasks described 
above (Gevins & Smith, 1999). Utilizing EEG fea- 
tures in the alpha and theta bands, average test set 
classification accuracy was 92% (range 84-100%, 
average binomial p < .001). In another study, we 
explicitly compared metrics based on either behav- 
ioral response measures during an n-back WM 
task, EEG recordings during task performance and 
control conditions, or combinations of behavioral 
and EEG variables with respect to their relative 
sensitivity for discriminating conditions of drowsi- 
ness associated with sleep loss from alert, rested 
conditions (Smith et al., 2002). Analyses based 



Electroencephalography (EEG) in Neuroergonomics 



21 



on behavior alone did not yield a stable pattern 
of results when viewed over test intervals. In con- 
trast, analyses that incorporated both behavioral 
and neurophysiological measures displayed a 
monotonic increase in discriminability from alert 
baseline with increasing amounts of sleep depri- 
vation. Such results indicate that fairly modest 
amounts of sleep loss can induce neurocognitive 
changes detectable in individual subjects perform- 
ing computer-based tasks, and that the sensitivity 
for detecting such states is significantly improved 
by the addition of EEG measures to behavioral 
indices. 



Extension of EEC-Based Cognitive 
State Monitoring Methods to More 
Realistic Task Conditions 

The results described above provide evidence 
for the basic feasibility of using EEG-based meth- 
ods for monitoring cognitive task load, mental fa- 
tigue, and drowsiness in individuals engaged in 
computer-based work. However, the n-back WM 
task makes minimal demands on perceptual and 
motor systems, and only requires that a subjects 
effort be focused on a single repetitive activity. In 
more realistic work environments, task demands 
are usually less structured and mental resources of- 
ten must be divided between competing activities, 
raising questions as to whether results obtained 
with the n-back task could generalize to such con- 
texts. 

Studies have demonstrated that more com- 
plicated forms of human-computer interaction 
(such as videogame play) produce mental effort- 
related modulation of the EEG that is similar to 
that observed during n-back tasks (Pellouchoud, 
Smith, McEvoy, & Gevins, 1999; Smith, McEvoy, & 
Gevins, 1999). This implies that it might be pos- 
sible to extend EEG-based multivariate methods 
for monitoring task load to such circumstances. 
To evaluate this possibility, a subsequent study 
(Smith et al., 2001) was performed in which the 
EEG was recorded while subjects performed the 
Multi- Attribute Task Battery (MATB; Comstock 
& Amegard, 1992). The MATB is a personal 
computer-based multitasking environment that 
simulates some of the activities a pilot might be re- 
quired to perform. It has been used in several prior 
studies of mental workload and adaptive automa- 



tion (e.g., Fournier, Wilson, & Swain, 1999; Para- 
suraman, Molloy, & Singh, 1993; Parasuraman, 
Mouloua, & Molloy, 1996). The data collected dur- 
ing performance of the MATB were used to test 
whether it is possible to derive combinations of 
EEG features that can be used for indexing task 
loading during a relatively complex form of human- 
computer interaction. 

The MATB task included four concurrently 
performed subtasks in separate windows on a com- 
puter screen (for graphic depictions of the MATB 
visual display, see Fournier et al, 1999; Molloy &r 
Parasuraman, 1996). These included a systems- 
monitoring task that required the operator to mon- 
itor and respond to simulated warning lights and 
gauges, a resource management task in which fuel 
levels in two tanks had to be maintained at a cer- 
tain level, a communications task that involved 
receiving audio messages and making frequency 
adjustments on virtual radios, and a compensatory 
tracking task that simulated manual control of air- 
craft position. Manipulating the difficulty of each 
subtask served to vary load; such manipulations 
were made in a between-blocks fashion. Subjects 
learned to perform low-, medium-, and high-load 
(LL, ML, and HL) versions of the tasks. For com- 
parison purposes they also performed a passive 
watching (PW) condition in which they observed 
the tasks unfolding without actively performing 
them. 

Subjects engaged in extensive training on the 
tasks on one day, and then returned to the labora- 
tory on a subsequent day for testing. On the test 
day, subjects performed multiple 5-minute blocks 
of each task difficulty level. Behavioral and subjec- 
tive workload ratings provided evidence that on 
average workload did indeed increase in a monoto- 
nic fashion across the PW, LL, ML, and HL task 
conditions. This increase in workload was associ- 
ated with systematic changes in the EEG. In partic- 
ular, as in the prior study of workload changes in 
the n-back task paradigm, frontal theta band activ- 
ity tended to increase with increasing task difficulty, 
whereas alpha band activity tended to decrease. 
Such results indicated that the workload manipula- 
tions were successful, and that spectral features in 
the theta and alpha range might be useful in at- 
tempting to automatically monitor changes in 
workload with EEG measures. 

Separate blocks of data were thus used to derive 
and then independently validate subject-specific, 
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Figure 2.2. Mean and SEM (N = 16 subjects) EEG-based cognitive workload 
index values during performance of the MATB flight simulation task. Data are 
presented for each of four task versions (PW = passive watch, LL = low load, 
ML = moderate load, HL = high load). Average cognitive workload index 
scores increased monotonically with increasing task difficulty. Data from 
Smith., Gevins, Brown, Karnik and Du (2001). 



EEG-based, multivariate cognitive workload func- 
tions. In contrast to the two-class pattern detection 
functions that were employed to discriminate be- 
tween different task load levels in the prior study, 
we evaluated a different technique that results in a 
single subject-specific function that produces a 
continuous index of cognitive workload and hence 
could be applied to data collected at each difficulty 
level of the task. In this procedure, the EEG data 
were first decomposed into short windows and a 
set of spectral power estimates of activity in the 
theta and alpha frequency ranges was extracted 
from each window. A unique multivariate function 
was then defined for each subject that maximized 
the statistical divergence between a small sample of 
data from low and high task load conditions. To 
cross-validate the function, it was tested on new 
data segments from the same subject. Across sub- 
jects (figure 2.2), mean task load index values were 
found to increase systematically with increasing 
task difficulty and differed significantly between the 
different versions of the task (Smith et al., 2001). 
These results provide encouraging initial evidence 
that EEG measures can indeed provide a modality 
for measuring cognitive workload during more 
complex forms of computer interaction. Although 
complex, the signal processing and pattern classifi- 
cation algorithms employed in this study were de- 
signed for real time implementation. In fact, a 
prototype online system running on a circa 1997 
personal computer performed the requisite calcula- 
tions online and provided an updated estimate of 



cognitive workload at 4-second intervals while 
subjects were engaged in task performance. 

It is worth reiterating here the critical role that 
effective automated artifact detection and filtering 
plays in such analyses. Effective artifact detection 
and filtering is particularly important during com- 
plex computer-based activities such as videogame 
play, as these types of behaviors tend to be associ- 
ated with a great deal of artifact-producing head, 
body, and eye movement that might confound 
EEG-derived estimates of cognitive state. For ex- 
ample, figure 2.3 illustrates the average workload 
indices obtained from data from a single electrode 
(frontal central site Fz) in an individual subject 
during the MATB, obtained after calibrating a mul- 
tivariate index function for that electrode using 
artifact-decontaminated examples of data from the 
low-load and high-load MATB conditions and then 
applying the resulting function to new samples of 
EEG data that were either decontaminated with 
state-of-the-art EEG artifact detection and filtering 
algorithms (leftmost and center columns) or with- 
out systematic artifact detection and correction 
(rightmost column), with N=50 4-second index 
function scores per task condition. A linear dis- 
criminant function applied to the data was able 
to correctly discriminate 95% of the individual 
clean samples of LL MATB data as coming from 
that category rather than from the HL category 
(binomial p < .000001). In contrast, an equivalent 
linear discriminant function applied to the artifact- 
contaminated LL data performed at chance level. 
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Figure 2.3. Individual subject workload index function scores from a single EEG 
channel (frontal central electrode Fz) can discriminate low from high load levels 
during MATB task performance when effective EEG artifact decontamination is 
employed (left and center columns), but load can be misclassified without such 
correction (right column). 



Analogous methods have also been used in a 
small exploratory study that involved more natura- 
listic computer tasks. In that experiment (Gevins & 
Smith, 2003), the EEG data were recorded while 
subjects performed more common computer-based 
tasks that were performed under time pressure and 
that were more or less intellectually demanding. 
These more naturalistic activities required subjects 
to perform word processing, take a computer-based 
aptitude test, and search for information on the 
Web. The word processing task required subjects to 
correct as many misspellings and grammatical er- 
rors as they could in the time allotted, working on a 
lengthy text sample using a popular word pro- 
cessing program. The aptitude test was a practice 
version of the Computer-Adaptive GMAT test. Sub- 
jects were asked to solve as many data-sufficiency 
problems as possible in the time allotted; such prob- 
lems make a high demand on logical and quantita- 
tive reasoning skills and require significant mental 
effort to complete in a timely fashion. The Web- 
searching task required subjects to use a popular 
Web browser and search engine to find as many an- 
swers as possible in the time allotted to a list of 
trivia questions provided by the experimenter. For 
example, subjects were required to use the browser 
and search engine to "convert 98.6 degrees Fahren- 
heit into degrees Kelvin," "find the population of 
the 94105 area code in the 1990 U.S. Census," 
and "find the monthly mortgage payment on a 
$349,000, 30-year mortgage with a 7.5% interest 



rate." Each type of task was structured such that 
subjects would be unlikely to be able to complete it 
in the time allotted. Data were also recorded from 
subjects as they performed easy and difficult n-back 
working memory tasks, and as they rested quietly, 
for comparison with the more naturalistic tasks. 

The same basic analysis procedure described 
above that was applied to the EEG data recorded 
during MATB performance was also employed in 
this study to derive personalized continuous func- 
tions indexing cognitive workload. The resulting 
functions were then applied to new samples of that 
subject's data. 

A summary of the results from these analyses, 
averaged across data segments within each task con- 
dition and compared between conditions, is pre- 
sented in figure 2.4. These comparisons indicate 
that the cognitive load index performed in a pre- 
dictable fashion. That is, the condition in which the 
subject was asked to passively view a blank screen 
produced an average EEG-based cognitive workload 
load around the zero point of the scale. Average in- 
dex values during 0-back task performance were 
slightly higher than those during the resting condi- 
tion, and average index values during the 3-back 
task were significantly higher than those recorded 
either during the 0-back WM task or during the 
resting state. All three naturalistic tasks produced 
workload index values slightly higher than those ob- 
tained in the 3-back task, which might be expected 
given that the n-back tasks had been practiced and 
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Figure 2.4. Mean and 5EM (N = 7 subjects) EEG-based cognitive workload index values dur- 
ing resting conditions, easy and difficult versions of the n-back working memory tasks, and a 
few naturalistic types of computer-based work (see text for full description of tasks and proce- 
dure). The data represent average index values over the course of each type of task. The easy 
WM and resting conditions produced significantly lower values than the more difficult WM 
condition or the naturalistic tasks. 



were repetitive in nature, whereas the other tasks 
were novel and required the use of strategies of in- 
formation gathering, reasoning, and responding that 
were less stereotyped in form. Among the naturalis- 
tic tasks, the highest levels of cognitive workload 
were recorded during the computerized aptitude- 
testing task — the condition that was also subjec- 
tively experienced as the most difficult. 

This pattern of results is interesting not only be- 
cause it conforms with a priori expectations about 
how workload would vary among the different 
tasks, but also because it provides data relevant to 
the issue of how the workload measure is affected by 
differences in perceptuomotor demands across con- 
ditions. Since in the n-back tasks stimuli and motor 
demands are kept constant between the 0-back and 
3-back load levels, the observed EEG differences in 
those conditions are clearly closely related to differ- 
ences in the amounts of mental work demanded by 
the two task variants rather than other factors. How- 
ever, in the study of MATB task performance de- 
scribed above, the source of variation in the index is 
somewhat less clear. On the one hand, performance 
and subjective measures unambiguously indicated 
that the mental effort required to perform the high- 
load version of the MATB was substantially greater 
than that required by the low-load (or passive 



watching) versions. On the other hand, the percep- 
tuomotor requirements in the high-load version 
were also substantially greater than those imposed 
by the other version. In this latter experiment, such 
confounds were less of a concern. Indeed, both the 
text editing task and the Web searching task re- 
quired more effortful visual search and more active 
physical responding than the aptitude test, whereas 
the aptitude test had little reading and less respond- 
ing and instead required a great deal of thinking 
and mental evaluation of possibilities. Thus, the fact 
that the average cognitive workload values during 
performance of the aptitude test were higher than 
those observed in the other tasks provides conver- 
gent support for the notion that the subject-specific 
indices were more closely tracking variations in 
mental demands rather than variations in percep- 
tuomotor demands in these instances. Nevertheless, 
the results remain ambiguous in this regard. 



From Unidimensional to 
Multidimensional Neurophysiological 
Measures of Workload 

Another approach to resolving the inherent ambi- 
guity of the sort of unidimensional "whole brain" 
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metric used to quantify mental workload in the 
studies described above is to generalize the metric 
to separate index loading of different functional 
brain systems. That is, the applied psychology and 
ergonomics literature has long posited a relative in- 
dependence of the resources involved with higher- 
order executive processes and those involved with 
perceptual processing and motor activity (Gopher, 
Brickner, & Navon, 1982; Wickens, 1991). Fur- 
thermore, related topographic differences can be 
observed in regional patterns of EEG modulation. 
For example, it is clear that alpha band activity 
over posterior regions is particularly sensitive to vi- 
sual stimulation and that increases in motor de- 
mands are associated with suppression of alpha 
and beta band activity over sensorimotor cortex 
(Arroyo et al., 1993; Jasper & Penfield, 1949; Mul- 
holland, 1995). Such regional differences can also 
be observed during performance of complex tasks. 
In one study, the EEG was recorded from subjects 
while they either actively played a videogame or 
watched the screen while someone else played the 
game (Pellouchoud et al, 1999). Across the group 
of subjects, the amplitude of the frontal midline 
theta rhythm was larger in the active performance 
condition than in the resting or passive watching 
conditions. In contrast, a posterior alpha band sig- 
nal was attenuated during both the playing and the 
watching conditions relative to the resting condi- 
tion, suggesting that it was responding primarily to 
the presence of complex visual stimulation rather 
than active task performance. Finally, a central mu 
(10-13 Hz) rhythm recorded over sensorimotor 
cortex was attenuated during the active game- 
playing condition, but not during the passive 
watching condition, presumably reflecting activa- 
tion related to the game's hand and finger motor 
control requirements (Pellouchoud et al., 1999). In 
another study where subjects were allowed to prac- 
tice a videogame until they were skilled at it, the al- 
pha rhythm recorded over frontal regions increased 
in amplitude with progressive amounts of practice, 
suggesting that smaller neuronal populations were 
required to regulate attention as the task became 
automated. In contrast, the alpha rhythm recorded 
over posterior regions displayed no such effect, 
suggesting that neural activation related to visual 
processing did not diminish (Smith et al., 1999). 

Such considerations have led to an extension 
of the method described above to create multidi- 
mensional indices that provide information about 



the relative activation of a local neocortical region. 
In particular, instead of defining a single load- 
sensitive multivariate function for the whole head, 
we have worked toward extracting three indepen- 
dent topographically regionalized metrics from mul- 
tielectrode data (Smith &r Gevins, 2005) recorded in 
the MATB experiment described above. One metric 
was derived from data recorded over frontal corti- 
cal areas. Since this region of the brain is known to 
be involved in executive attention control and 
working memory processes, we refer to this metric 
as a measure of cortical activation related to frontal 
executive workload. A second metric was derived 
from data recorded from central and parietal re- 
gions. Since these regions are activated by motor 
control functions, somatosensory feedback, and 
the coordination of action plans with representa- 
tions of extra personal space, we refer to this second 
metric as a measure of sensorimotor activation. A 
third metric was derived from electrodes over oc- 
cipital regions. Since this region includes primary 
and secondary visual cortices, we refer to this third 
metric as representing variation in cortical activa- 
tion due to visuoperceptual functions. While these 
labels are convenient for discussion, they of course 
are highly oversimplified with regard to describing 
the actual operations performed by the underlying 
cortical systems. They may, however, be seen as 
consistent with the results of fMRI studies of simu- 
lator (driving) operation (Calhoun et al., 2002; 
Walter et al., 2001), which have also reported acti- 
vation in frontal attention networks, sensorimotor 
cortex, and visual cortices (see also chapter 4, this 
volume). 

Figure 2.5 summarizes how the three regional 
cortical activation metrics changed as a result of 
task manipulations, describing the mean output of 
the regional metrics computed across all of the 
cross-validation data segments for each task diffi- 
culty level for each subject. Each regional metric 
was found to be significantly affected by the task 
difficulty manipulation, consistent with the notion 
that the MATB task increased workload on multi- 
ple brain systems in parallel. Furthermore, both 
subjective workload estimates and overt task per- 
formance were found to covary with the regional 
EEG-derived workload estimates, indicating the 
metrics were tracking changes in brain activity that 
were functionally significant. 

In a second experiment, these regional work- 
load metrics were tracked over the course of an 
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Figure 2.5. Average normalized (SEM) values for regional cortical activation metrics over 
frontal, central, and posterior regions oi the scalp derived from multivariate combinations of 
EEG spectral features recorded from N = 16 participants performing 5-minute blocks of 
high-, medium-, and low-load versions of the MATB task. 



all-night experiment during which subjects per- 
formed the HL version of the MATB and other 
tasks in a more or less continuous fashion without 
sleeping since early the prior morning (Smith & 
Gevins, 2005; Smith et al., 2002). During this ex- 
tended wakefulness session, cortical activation as 
indexed by the regional EEG workload scores was 
observed to change with time on task despite task 



difficulty being held constant and despite the fact 
that subjects were highly practiced in the task. The 
changes are illustrated in figure 2.6. The daytime 
values are contrasted with values representing the 
first block of data from the overnight session, 
where testing on average began around 11:00 p.m. 
They are also contrasted with late night values 
from the time period within the last four test inter- 
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Figure 2.6. Average normalized (_SEM) values for regional cortical activation metrics 
over frontal, central, and posterior regions of the scalp derived from multivariate com- 
binations of EEG spectral features recorded from N = 16 participants performing 
5-minute blocks of the high-load MATB task during alert daytime baseline periods 
(around noon or 1200 hrs), during the first test interval of an all-night recording ses- 
sion (2300 hrs or 11:00 p.m.), and during the test interval between 1:30 and 5:30 a.m. 
in which they displayed a cortical activation minima (across subjects this minimum 
occurred on average at 0400 hrs or 4:00 a.m.). 
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vals for each subject when he or she displayed a 
minimum in total cortical activation (for 15/16 
subjects this minimum occurred between 1:30 and 
5:30 a.m.). Average values for each region declined 
with sleep deprivation, with the largest overall de- 
clines for the frontal region. 

Interestingly, subjective workload was found 
to be negatively correlated with the magnitude of 
the fatigue-related decline in the frontal region — 
but not the other regions — suggesting that as 
frontal activation decreased the subjects found it 
increasingly difficult to confront the demands of 
the high-load MATB task. The fact that perceived 
mental effort was observed to be positively corre- 
lated with changes in frontal cortical activity in 
alert individuals, yet negatively correlated with 
frontal cortical activation with increases in mental 
fatigue, might be seen as problematic for the even- 
tual development of adaptive automation systems 
that aim to dynamically modulate the cognitive 
task demands placed on an individual in response 
to momentary variations in the availability of 
mental resources as reflected by real-time analysis 
of neural activity. That is, it has sometimes been 
suggested that it might be possible to use mea- 
sures of brain activation as a basis for automated 
systems to off-load tasks from an individual if he 
or she was detected to be in a state of high cogni- 
tive workload, or allocate more tasks to an indi- 
vidual that appeared to have ample reserve 
processing capacity and was in danger of becom- 
ing bored or inattentive. The current results indi- 
cate that a decrease in cortical activation in frontal 
regions may reflect either a decrease in mental 
workload or an increase in mental fatigue and a 
heightened sense of mental stress. Assigning more 
tasks to an individual in the former case may in- 
deed serve to increase his or her cognitive 
throughput. In the latter case, it may result in the 
sort of tragic accident that is too often reported to 
occur when fatigued personnel are confronted 
with unexpected increases in task demands 
(Dinges, 1995; Miller, 1996; Rosekind, Gander, & 
Miller, 1994). Thus, while measures of brain func- 
tion during complex task performance may serve 
to accelerate research into the sources of perfor- 
mance failure under stress, it seems likely that a 
great deal of future research will be needed before 
such measures can be adapted to the problem of 
developing technology for adaptively augmenting 
the capabilities of mission-critical personnel work- 



ing in demanding and stressful computerized-task 
environments. 



Conclusion 

In summary, the results reviewed above indicate that 
the EEG changes in a highly predictable way in re- 
sponse to sustained changes in task load and associ- 
ated changes in the mental effort required for task 
performance. It also changes in a reliable fashion in 
response to variations in mental fatigue and level of 
arousal. It appears that such changes can be auto- 
matically detected and measured using algorithms 
that combine parameters of the EEG power spectra 
into multivariate functions. While such EEG metrics 
lack the three-dimensional spatial resolution pro- 
vided by neuroimaging methods such as PET or 
fMRI, they can nonetheless provide useful informa- 
tion about changes in regional functional brain sys- 
tems that may have important implications for 
ongoing task performance. Such methods can be ef- 
fective both in gauging the variations in cognitive 
workload imposed by highly controlled laboratory 
tasks and in monitoring differences in the mental ef- 
fort required to perform tasks that more closely re- 
semble those that an individual might encounter in 
a real-world work environment. Because this sensi- 
tivity can in principle be obtained with technology 
suitable for use in real- world work environments, 
the EEG can be seen as a critical tool for research in 
neuroergonomics . 
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Main Points 

1 . The EEG recorded at the scalp is a record of 
instantaneous fluctuations of mass electrical 
activity in the brain, primarily summated 
postsynaptic (dendritic) potentials of large 
cortical neuronal populations. 

2. Spectral components of the EEG signal show 
characteristic changes in response to variations 
in mental demands or state of alertness. As 
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with all other means of measuring brain 
function, EEG signals are also sensitive to the 
perceptual and motor activities of the subject 
in addition to mental activity. It is essential to 
separately measure these perceptual and 
motoric neural processes to have a strong 
inference that the brain function signals one 
would like to use as a measure of mental 
activity actually in fact do so. 

3. The high temporal resolution of the EEG in 
combination with the simplicity and 
portability of the technology used to record 
and analyze it make it suitable for use in 
unrestrained subjects in a relatively wide range 
of environments, including real-world work 
contexts. 

4. The sensitivity of EEG signals to particular 
task demands differs depending on the spatial 
positioning of scalp electrodes and, in many 
but not all cases, reflects functional 
specialization of nearby underlying cortical 
regions. 

5. As with all brain function measurement 
technologies, the EEG signal is sensitive to 
artifactual contaminants not generated in the 
brain, which must be removed from the signal 
in order to make valid inferences about mental 
function. This is easier said than done. 

6. There is no simple one-to-one mapping 
between a change in a measure of brain 
activation and the cognitive loading of an 
individual. Additional factors, such as the state 
of alertness, must be taken into account. 
Simplistic approaches to neuroadaptive 
automation that do not take this complexity 
into account will fail. 
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Event-Related Potentials (ERPs) 
in Neuroergonomics 



Event-related potentials (ERPs) represent the brain's 
neural response to specific sensory, motor, and cog- 
nitive events. ERPs are computed by recording the 
electroencephalogram (EEG) from the scalp of a 
human participant and by averaging EEG epochs 
time-locked to a particular event. The use of ERPs 
to examine various aspects of human cognitive 
processes has a long history. Pioneering work on 
ERP correlates of cognitive processes such as atten- 
tion (Hillyard, Hink, Schwent, & Picton, 1973), 
working memory (Donchin, 1981), and language 
(Kutas & Hillyard, 1984) were carried out in the 
1970s and 1980s. These studies were important 
because they established the use of ERPs as a tool 
for mental chronometry (Posner, 1978), or the ex- 
amination of the timing of the neural events associ- 
ated with different components of information 
processing. However, these landmark studies did 
not greatly influence theory or empirical research 
in cognitive psychology in the era in which they 
were carried out. Moreover, because of their poor 
spatial resolution in localizing sources of neuronal 
activity underlying scalp electrical potentials, ERPs 
were not well regarded by neuroscientists accus- 
tomed to the spatial precision of single-cell record- 
ing in animals. The mid-1980s were a period when 
the cognitive neuroscience revolution was in its 



early phases (Gazzaniga, 1995). Consequently, 
ERP research did not enjoy much currency in the 
mainstream of either cognitive psychology or neu- 
roscience. 

The situation changed a few years later. The 
development of other neuroimaging techniques 
such as positron emission tomography (PET) and 
functional magnetic resonance imaging (fMRI) led 
to their growing use to examine the neural basis of 
human cognitive processes, beginning with the 
seminal work of Posner, Petersen, Fox, and Raichle 
(1988). Neuroimaging allowed for the rediscovery 
of ERPs in cognitive psychology and cognitive neu- 
roscience. As a result, ERPs made a comeback in 
relation to both psychology and neuroscience and 
today enjoy an acknowledged status in both fields. 
The importance of ERPs as a tool in cognitive neu- 
roscience was further enhanced with the realiza- 
tion that PET, fMRI, and related neuroimaging 
techniques had serious limitations in their tempo- 
ral resolution of assessing neural processing, de- 
spite their great advantage over ERPs with respect 
to spatial localization of neuronal activity. 

At the present time, therefore, ERPs hold a 
unique position in the toolshed of cognitive neu- 
roscientists. Because of the inherent sluggishness 
(several seconds) of neuroimaging techniques (PET 
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and fMRI) based on cerebral hemodynamics 
compared to the time scale of neuronal activity 
(milliseconds), ERPs are being increasingly used 
whenever there is a need to examine the relative 
timing of neural mechanisms underlying cogni- 
tive processes. In a number of cases, the timing 
information provided by ERPs has been critical in 
resolving a major theoretical issue in cognitive sci- 
ence. Probably the best-known example involves 
the use of ERPs to address the early versus late 
selection debate in selective attention research 
(Hillyard et al., 1973; Luck & Girelli, 1998; Man- 
gun, 1995). 

ERP research has also been applied to practical 
issues in a number of domains, most notably in neu- 
rology and psychiatry. In general, most applications 
of ERPs have involved the investigation of abnormal 
behavior, as in ERP studies of neuropsychiatric con- 
ditions such as schizophrenia and Alzheimer's dis- 
ease. ERP studies of normal populations have also 
been carried out, but primarily in relation to issues 
of child development and normal aging. In contrast, 
applications (as opposed to basic research) of ERPs 
to problems of human performance in normal 
everyday situations have been relatively infrequent. 
Nevertheless, there is a small but noteworthy lit- 
erature on ERP applications to problems of human 
performance in work situations — the province of 
human factors or ergonomics. 

Applications of ERP to issues relevant to hu- 
man factors and ergonomics involve a number of 
topics. These include the assessment of mental 
workload, evaluation of mechanisms of vigilance 
decrement, and monitoring of operator fatigue in 
human-machine systems. Other areas include the 
use of ERPs to assess the influence of stressors, au- 
tomation, and online adaptive aiding on human 
operator performance. These studies have been re- 
viewed elsewhere (see Byrne & Parasuraman, 
1996; Kramer & Belopolsky 2003; Kramer & We- 
ber, 2002; Parasuraman, 1990; Wickens, 1990) and 
so are not revisited here. Rather, the purpose of this 
chapter is to provide a methodological overview 
of the use of ERPs and their potential application 
to the examination of issues in human factors and 
ergonomics. 

In this chapter, we describe how ERPs can be 
recorded and analyzed, and discuss research on a 
number of ERP components, focusing on those 
that are particularly relevant to human factors 
issues: the early-latency attention- related PI and 



Nl; the long-latency P3 or P300; the mismatch 
negativity (MMN); the lateralized readiness poten- 
tial (LRP); and the error-related negativity (ERN). 
We discuss the use of these ERP components to ad- 
dress four neuroergonomic issues: (1) assessment 
of mental workload; (2) understanding the neural 
basis of error detection and performance monitor- 
ing; (3) response readiness; and (4) studies of auto- 
matic processing. We begin, however, by outlining 
the specific advantages offered by ERPs in neuroer- 
gonomics research. 



ERPs in Relation to Other 
Neuroimaging Techniques 

A core feature of neuroergonomics is an interest in 
brain mechanisms in relation to human perfor- 
mance at work (Parasuraman, 2003). To this end, 
researchers may make use of physiological measures 
that reflect, more or less directly, aspects of brain 
function. Alternatively, neuroergonomic research 
may not directly involve such measures but rely on 
the results of studies of brain function conducted by 
others to guide hypotheses regarding the design of 
human-machine interfaces or the training of opera- 
tors of such systems. When physiological measures 
are used, the most direct are those derived from the 
brain itself: the EEG, which represents the sum- 
mated activity of dendritic (postsynaptic) popula- 
tions of neuronal cells as recorded on the scalp; 
magnetoencephalogaphy (MEG), which consists of 
the associated magnetic flux that is recorded at the 
surface of the head; and ERPs and event-related 
magnetic fields, which constitute the brains specific 
response to sensory, cognitive, and motor events. In 
addition to these electromagnetic measurements, 
measures of the brains metabolic and vascular re- 
sponse (PET and fMRI), which can be linked to 
neuronal activity, also provide a noninvasive win- 
dow on human brain function. 

Currently, electromagnetic measures such as 
ERPs provide the best temporal resolution (1 ms 
or better) for evaluating neural activity in the hu- 
man brain, and metabolic measures such as fMRI 
have the best spatial resolution (1 cm or better). 
No single technique combines both high temporal 
and spatial resolution. Furthermore, some of these 
techniques (e.g., fMRI) are expensive and restrict 
participant movement, which makes them difficult 
to use for neuroergonomic studies. However, new 
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imaging technologies are being developed, such 
as functional near-infrared spectroscopy (fNIRS) 
and other forms of optical imaging, that promise to 
provide high temporal and spatial resolution (Grat- 
ton & Fabiani, 2001). These techniques have the 
additional advantage of being more portable and 
less expensive than fMRI, which will therefore add 
them to the catalog of available methods that are 
appropriate for neuroergonomic research. (For a 
review of brain imaging techniques and their appli- 
cation to psychology, see Cabeza & Kingstone, 
2001; see also chapters 4 and 5, this volume). 



Fundamentals of ERPs 

ERPs are recorded using sensors (tin or silver elec- 
trodes) placed on the scalp of human participants 
and by extracting and signal averaging samples of 
the EEC The ERP represents the average of the 
EEG samples that are time-locked to a particular 
event. The time-locking event can be an external 
stimulus (e.g., sounds, words, faces, etc.), the be- 
havioral response elicited by the participant (e.g., a 
button press, speech, or other motor movement), 
or an internal cognitive operation such as (silent) 
identification or making a decision choice. The sig- 
nal averaging can be done either forward or back- 
ward in time with respect to the time-locking 
event; typically backward averaging is done for 
response-related ERPs. For time-locking to an in- 
ternal, unobservable cognitive event, some external 
temporal marker indicating when that event is 
likely to have occurred is necessary in order to ob- 
tain an ERP. 

The signal averaging technique assumes that 
any EEG activity that is not time-locked to the 
event sums to zero if sufficient numbers of samples 
are taken, and that the resulting ERP waveform re- 
flects the brains specific response to the eliciting 
event. This assumption has been questioned over 
the years, as there is evidence that EEG compo- 
nents (such as alpha desynchronization) do not 
cancel out with repeated averaging. There is also 
evidence that such event-related EEG responses 
may carry important information regarding percep- 
tual or cognitive processing (Makeig et al., 2002). 
However, a review of this research is beyond the 
scope of this chapter, in which we focus on the use 
of specific ERP components in relation to neuroer- 
gonomic issues. 



The amplitude of the ERP is usually on the or- 
der of about several microvolts. This is smaller than 
that of the background EEG that is recorded during 
resting conditions, which tends to be in the range of 
tens of microvolts. Because the ERP is time-locked 
to events that are embedded into the noisy back- 
ground of EEG, signal averaging over several 
samples is necessary. Depending on the ERP com- 
ponent of interest, typically tens or hundreds of 
samples need to be included in the average. 

Signal averaging of the EEG must be accompa- 
nied by recording of eye and body movements, be- 
cause the electrical signals from these sources can 
contaminate the EEG. For a better estimation of the 
true neural responses to the time-locked stimulus, 
participant motor action, or cognitive event, EEG 
epochs with eye blinks or body movements need to 
be rejected before averaging. The difficulties, par- 
ticularly in field settings, of obtaining artifact-free 
samples of EEG are frequently underestimated. 
Consequently, reliable correction of EEG for arti- 
facts is a prerequisite for use of ERPs in neuroer- 
gonomics. The magnitude of the electrooculogram 
(EOG) is typically used for artifact correction. The 
EOG is recorded from electrodes about 1 cm out- 
side the canthi (near the eyes) and from electrodes 
above and below the left or right eye to monitor 
horizontal eye movements and blinks. If muscle 
movements occur, these can generally be seen as 
large, high-frequency activity and can be relatively 
easily recognized in the raw EEG waveform. An al- 
ternative to rejection of a particular EEG epoch 
that is contaminated with eye movement is to use a 
subtraction technique prior to signal averaging. In 
this method, a model-estimated contribution of the 
EOG to the EEG epoch is subtracted from the EEG 
sample, and that sample is retained in the average 
(Gratton, Coles, & Donchin, 1983). However, this 
method requires an extra assumption of the relia- 
bility of the EOG correction model, and so a more 
conservative method is to simply reject all trials 
with excessive EOG and to increase the overall 
number of samples to improve the signal-to-noise 
ratio (SNR) of the ERP 

Obtaining an ERP waveform with a good SNR 
is a key methodological issue in ERP research. The 
best recordings occur in an electrically shielded and 
soundproofed room. However, this is not essential, 
and ERPs have been recorded in nonshielded envi- 
ronments and even in the field. An important re- 
quirement is to maintain the impedance of the 
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Figure 3.1 . Illustration of the relationship between the signal-to-noise ratio (SNR) and 
the number of trials used for averaging an ERP The SNR is progressively improved, or 
the ERP waveforms are made less noisy, by increasing the number of trials for averaging 
from 16 to 32, 64, and 128. The single-trial data were averaged across EEGs elicited by a 
visual stimulus (four dots with each on the corner of a virtual rectangle) presented in the 
left visual field. The recording electrode, the right occipital site, overlies the occipitotem- 
poral area contralateral to the stimulus visual field. Data from one participant from Fu, 
Caggiano, Greenwood, and Parasuraman (2005). 



scalp electrodes below 5,000 ohms and to use low- 
noise, high-impedance amplifiers (Picton et al, 
2000). In theory, the SNR of an ERP is directly pro- 
portional to the square root of the number N of tri- 
als in the average, or vN. What this function 
indicates is that doubling the SNR requires that N 
should be quadrupled. For example, obtaining 
twice the SNR for an ERP with 16 trials would 



require 64 trials. Figure 3.1 shows how the SNR of 
an averaged ERP waveform can be improved by in- 
creasing the number of trials N. 

In practice, stable ERPs with adequate SNR 
can be obtained without an excessively large num- 
ber of trials, assuming that artifact-free trials are av- 
eraged, but the precise number of trials depends on 
the ERP component of interest. Roughly speaking, 
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Figure 3.2. Illustration of the visual ERP components PI, Nl, P2, 
N2, and P3 in response to the standard (thin line) and target (thick 
line) stimuli. Stimuli were arrows pointing to the northeast or 
northwest that were flashed randomly to the left and right visual 
field. Participants were asked to respond to one type of arrow (tar- 
gets, 10%) on both sides and to make no response to the other 
(standards, 90%). The size of stimuli could be large or small. The 
cues, which were presented 100 to 300 ms before the stimuli, 
could also be large or small. Data were averaged across 14 partici- 
pants and across stimulus size and cue validity. OR = right occipital 
site; OL = left occipital site. 



10-30 trials might be sufficient for an average of a 
large component like P3 in a laboratory recording 
situation, whereas 100 or even more trials might be 
needed for an average of relatively early and small 
components such as visual PI or CI (see below for 
definitions of these ERP components). However, it 
may not be practical to administer several hundred 
trials to a participant, because of the probability 
of participant fatigue and the resultant increased 
chance of muscle artifacts. Thus, a practical issue 
in designing an ERP study is to find a good balance 
between obtaining good SNR for the ERP compo- 
nents of interest and keeping participants alert and 
fatigue free. 

Once a stable, artifact-free ERP waveform with 
good SNR has been obtained, the peak amplitudes 
and latencies of specific ERP components — both 
positive and negative — can be measured. Typically, 
peak amplitude is measured relative to the averaged 
amplitude of a pre-event baseline (generally, 100 or 
200 ms before the onset of the event). Alternatively, 
the peak-to-peak amplitude difference between two 
consecutive positive and negative components, or 
the mean amplitude of a component, are sometimes 
reported. Figure 3.2 illustrates the ERPs elicited by 
a visual nontarget stimulus (thin line) and a target 
(thick line) stimulus in a selective attention para- 
digm. The ERP components of PI, Nl, P2, N2, and 
P3 are shown. 



The convention for naming ERP components 
has generally been based on the polarity (positive 
or negative) and the sequence or order of a spe- 
cific component in the ERP waveform. For exam- 
ple, PI is the first visual cortical response with a 
positive polarity and Nl is the first negative po- 
tential. Unfortunately, there are exceptions to this 
convention, which can occasionally cause some 
confusion. For example, in visual ERPs, there is a 
component that precedes PI, known as CI (for 
component 1, or the first visual ERP component). 
This component is generated in the striate cortex, 
prior to PI (thought to be generated in extrastriate 
cortex), and because the polarity of CI can be ei- 
ther positive or negative, depending upon the reti- 
nal location that is stimulated (Clark, Fan, & 
Hillyard, 1995; Clark & Hillyard, 1996). Another 
convention used to name ERP components is 
based on polarity and peak latency. For example, 
P70 represents a positive electrical response with a 
peak latency of 70 ms after stimulus onset, and 
PI 15 represents another positive component with 
a peak latency of 115 ms. This convention has the 
advantage of clarifying that the same component 
can have different latencies, because P70 and PI 15 
might both characterize the PI component. How- 
ever, the convention is also potentially misleading, 
because P180 could represent either a PI compo- 
nent for a slowly developing visual process (e.g., 
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object shape that is extracted from visual mo- 
tion) or a faster P2 component elicited by a flash- 
ing visual figure. Finally, some ERP components 
are derived by computing the difference in ERP 
waveforms between two conditions. In this case, a 
descriptive terminology is typically used. For ex- 
ample, the auditory MMN is the difference wave 
obtained by subtracting ERPs of standard (fre- 
quent) stimuli from ERPs of deviant (occasional) 
stimuli (Naatanen, 1990). Other ERP components 
using this descriptive nomenclature are the LRP 
and the ERN. 

ERPs can provide important information about 
the time course of mental processes, the brain re- 
gions that mediate these processes, and even com- 
munication between different brain areas. ERP 
patterns can be observed at different levels. At the 
single-electrode level, the amplitude, latency, and 
duration of different ERP components may vary 
across conditions. These differences may be com- 
pared across electrodes over different brain areas to 
get a clue as to the brain areas involved. At the 
two-dimensional scalp level, the scalp ERP voltage 
maps may vary in a specific time range when a cer- 
tain component of mental processing occurs. At a 
three-dimensional level, the location, strength, and 
orientation of the intracortical neural generator(s) 
corresponding to a specific mental processing stage 
may vary across conditions. 

ERPs are powerful for tracking the time course 
of mental processes, but not accurate in inferring 
the anatomy of the underlying neural activity, as 
compared with brain imaging techniques such as 
PET and fMRI. It can be misleading to infer from 
the observation of ERP activation at a particular 
scalp electrode site that the neural generator of that 
activity is located directly underneath the elec- 
trode. The reason is that neural activity generated 
in the brain is volume conducted and can be 
recorded at quite a distance from the source. For 
example, the so-called brain stem ERPs, which re- 
fer to a series of very short latency (<10 ms) ERP 
components following high-frequency auditory 
stimulation, and which are known from animal 
and lesion studies to be generated in the brain 
stem, are best recorded at a very distant scalp site, 
Cz, which is located at the top of the head. 

Source localization is a better solution to deter- 
mining the functional anatomy of neural activity as 
reflected in the scalp ERP A number of source local- 
ization methods have been proposed. Most of these 



methods try to solve the inverse problem — how to 
estimate the source of a volume-conducted signal 
from the distant pattern. One of the best known is 
called brain electrical source analysis (BESA; 
Scherg, 1990). In this method, successive forward 
solutions are projected in an attempt to solve the 
inverse problem: An initial location and orientation 
for a dipole (neural generator) are assumed, then 
the projected scalp pattern of that dipole is com- 
pared to the actual ERP pattern, and the error is 
used to correct the location and/or orientation of 
the dipole, with the process being repeated until 
the error is minimized. Unfortunately, this tech- 
nique faces the difficulty that no unique solution 
can be obtained while inferring a source from sur- 
face recordings. This is because any specific scalp 
voltage distribution can be caused by numerous 
configurations derived from different combinations 
of the number, orientation, and strength of neural 
generators in the brain. On the other hand, the 
distributed-solution-based LORETA algorithm (see 
figure 3.3) reveals the authentic but "blurred" 
three-dimensional point source with certain dis- 
persions at the activation location (Pascual-Marqui, 
1999), and thus, no accurate anatomic localization 
can be obtained. 

The temporal information that ERPs provide 
concerning the neural activity associated with cog- 
nitive processes is sufficient for many human 
factors applications. However, as discussed previ- 
ously, neuroergonomic research can also benefit 
from studies in which neural activity can be local- 
ized to specific brain locations, given that there 
may be other knowledge about the function of 
those brain regions to guide hypotheses relevant 
to issues in human factors problems. Hence, if 
ERPs can be combined with other imaging tech- 
niques with high spatial resolution, both temporal 
and localizing information might be gained. Ac- 
cordingly, several studies have combined ERP with 
PET or fMRI techniques that provide accurate 
anatomical information (Heinze et al., 1994; Man- 
gun, Hopfinger, Kussmaul, Fletcher, & Heinze, 
1997; Mangun et al., 2001). The localization in- 
formation provided by these imaging techniques 
can be used as seeds or start points for initial di- 
pole placement, instead of an arbitrarily chosen lo- 
cation. This constrained or seeded solution of ERP 
dipole modeling allows one to test the difference 
between the derived data from the seeded genera- 
tor and the observed scalp voltage data with high 
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Figure 3.3. Illustration of results obtained from LORETA (Pascual-Marqui, 1999), a distributed localization 
method. Data were for a left visual field target array under valid and invalid cue conditions, averaged across large 
and small cue conditions. The attentional effects on the PI and Nl range were obtained by subtracting ERPs of in- 
valid trials from ERPs of valid trials. The brain activations in response to these attentional effects are displayed at 
154 ms, when the global field power is maximal. It is clear that these attentional effects are distributed on the pos- 
terior brain regions contralateral to the stimulus visual field. That is, for the left visual field (LVF) stimulus, atten- 
tional effects are more pronounced in right posterior brain regions. See also color insert. 



temporal resolution. However, most studies using 
this combined approach have recorded neuroimag- 
ing and ERP data at separate times, although on 
the same group of participants and the same task. 
Therefore, an important issue for future research 
is to be able to simultaneously combine high- 
temporal-resolution ERP recording with high 
spatial-resolution imaging such as fMRI in the 
same subjects at the same time. 

There are a number of other methodological 
and experimental design issues related to the use 
of ERPs to examine the neural basis of cognitive 
processes. For space reasons, we have discussed 
only the most basic issues. For recent volumes that 
cover all the major methodological issues in ERP 
research, see Handy (2005) and Luck (2005). 



ERPs and Neuroergonomics 

ERPs can provide neural measures of the operators 
mental state with millisecond temporal resolution 
and a certain degree of spatial resolution. Such in- 
formation can be gleaned even when the operator 
does not make an overt behavioral response. This 
can be important under conditions when the oper- 
ator plays a supervisory role over automated systems. 
This type of information obtained from ERP record- 
ings is therefore specific and not available from 
other sources, such as expert judgment, biome- 
chanical measures, or motor responses. Of course, 
an important issue in applying ERPs to human fac- 
tors issues is that the ERP assessment must be reli- 
able, efficient, and valid (Kramer & Belopolsky, 
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2003). In the following, we discuss a number of 
different human factors issues where ERP studies 
have been conducted. 

Assessment of Mental Workload 

By far the largest number of ERP studies in human 
factors have addressed the issue of the assessment 
of mental workload. The topic continues to be of 
importance in human factors research because the 
design of an efficient human-machine system is not 
just a matter of assessing performance, but also of 
evaluating how well operators can meet the work- 
load demands imposed on them by the system. A 
major question that must be addressed is whether 
human operators are overloaded and can meet 
additional unexpected demands (Moray, 1979; 
Wickens, 2002). Behavioral measures such as ac- 
curacy and speed of response to probe events have 
been widely used to assess mental workload, but 
measures of brain function as revealed by ERPs of- 
fer some unique advantages that can be exploited 
in particular applications (Kramer & Weber, 2000; 
Parasuraman, 2003). Such measures can also be 
linked to emerging cognitive neuroscience knowl- 
edge on attention (Parasuraman & Caggiano, 2002; 
Posner, 2004), thereby allowing for the develop- 
ment of neuroergonomic theories of mental work- 
load. 

Of the ERP studies of mental workload, a large 
number have examined the P3 or P300 component, 
which was discovered by Sutton, Braren, Zubin, 
and John (1965). P300 is characterized by a slow 
positive wave with a mean latency following stimu- 
lus onset of about 300 ms, depending on stimulus 
complexity and other factors (Polich, 1987). The 
P300 typically has a centroparietal scalp distribu- 
tion, with a maximum over parietal cortex. P300 
amplitude is relatively large and therefore easily 
measured, sometimes even on single trials. The am- 
plitude of the P300 is very sensitive to the probabil- 
ity of presentation of the eliciting stimulus. In the 
typical oddball paradigm used to elicit the P300, 
two stimuli drawn from different perceptual or con- 
ceptual categories (e.g., red vs. green colors, low vs. 
high tones, or nouns vs. verbs) are presented in a 
random sequence. The stimulus that is presented 
with a low probability (e.g., 20%) elicits a larger 
P300 than the high-probability (e.g., 80%) stimu- 
lus. Since P300 amplitude is sensitive to the proba- 
bility of a task-defined category, it is thought to 



reflect postperceptual or postcategorical processes. 
In support of this view, increasing the difficulty of 
identifying the target stimulus (e.g, by masking) in- 
creases the latency of the P300 wave (Kutas, Mc- 
Carthy, & Donchin, 1977), whereas increases in the 
difficulty of response selection do not affect P300 la- 
tency (Magliero, Bashore, Coles, & Donchin, 1984). 
This has led to the view that the latency of the P300 
provides a relatively pure measure of perceptual 
processing and categorization time, independent of 
response selection and execution stages (Kutas et 
al., 1977; McCarthy & Donchin, 1981). Consistent 
with this view, P300 is sensitive to uncertainty both 
in detecting and identifying a masked target in noise 
(Parasuraman & Beatty, 1980). 

It has also been proposed that the amplitude of 
P300 is proportional to the amount of attentional 
resources allocated to the task (Johnson, 1986). 
Thus any diversion of processing resources away 
from target discrimination in a dual-task situation 
will lead to a reduction in P300 amplitude. For 
example, Wickens, Isreal, and Donchin (1977) 
showed that the amplitude of P300 decreased 
when a primary task, tone counting, was combined 
with a secondary task, visual tracking. However, 
increases in the difficulty of the tracking task (by 
increasing the bandwidth of the forcing function) 
did not lead to a further reduction in P300 ampli- 
tude (Isreal, Chesney, Wickens, & Donchin, 1980). 
This pattern of findings was taken to support the 
view that the P300 reflects processing resources as- 
sociated with perceptual processing and stimulus 
categorization, but not response-related processes, 
which were manipulated in the tracking task. In 
another study in which the tone-counting task was 
paired with a visual monitoring task whose percep- 
tual difficulty was manipulated (monitoring four 
vs. eight aircraft in a simulated air traffic control 
task), P300 amplitude was reduced with increases 
in difficulty. 

These and other related studies (Kramer, Wick- 
ens, & Donchin, 1983, 1985) have clearly estab- 
lished that P300 amplitude provides a reliable index 
of resource allocation related to perceptual and cog- 
nitive processing. Similar results have been obtained 
when P300 has been used in conjunction with per- 
formance in high-fidelity fixed-wing (Kramer, Sire- 
vaag, & Braune, 1987) and rotary-wing aircraft 
(Sirevaag et al., 1993). P300 therefore provides a 
reliable and valid index of mental workload, to the 
extent that perceptual and cognitive aspects of 
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information processing are major contributors to 
workload in a given performance setting. Response- 
related contributions to workload, however, are not 
reflected in P300 amplitude. 

Two recent examples of human factors studies 
that exploited these characteristics of P300 to as- 
sess cognitive workload are briefly described here. 
Schultheis and Jameson (2004) conducted a study 
of the difficulty of text presented in hypermedia 
systems, with a view to investigating whether the 
text difficulty could be adaptively varied depend- 
ent on the cognitive load imposed on the user. 
They paired an auditory oddball task with easy and 
difficult versions of text and measured pupil diam- 
eter and the P300 to the oddball task. They found 
that P300 amplitude, but not pupil diameter, was 
significantly reduced for the difficult hypermedia 
condition. The authors concluded that P300 am- 
plitude and other measures, such as reading speed, 
may be combined to evaluate the relative ease of 
use of different hypermedia systems. 

Baldwin, Freeman, and Coyne (2004) used a 
visual color-discrimination oddball task to elicit 
P300 during simulated driving under conditions of 
normal and reduced visibility (fog). P300 ampli- 
tude, but not discrimination accuracy or reaction 
time (RT), was reduced when participants drove in 
fog compared to driving with normal visibility. In 
contrast, P300 was not sensitive to changes in traf- 
fic density, while the behavioral measures were sen- 
sitive to this manipulation of driving difficulty. The 
authors concluded that neither neural nor behav- 
ioral measures alone are sufficient for assessing cog- 
nitive workload during different driving scenarios, 
but that multiple measures may be needed. They 
speculated that such combinatorial algorithms could 
be used to trigger adaptive automation during high 
workload by engaging driver-aiding systems and 
disengaging secondary systems such as cellular 
phones or entertainment devices. 

Resource allocation theories propose that the al- 
location of processing resources to one task in a 
dual-task pairing leads to a withdrawal of resources 
to the second task (Navon & Gopher, 1979; Wick- 
ens, 1984). The P300 studies showed that if the sec- 
ond task was an oddball task, then P300 amplitude 
was reduced. At the same time, resource allocation 
theories would predict an increase in P300 ampli- 
tude with the allocation of resources to the primary 
task. This was demonstrated by Wickens, Kramer, 
Vanesse, and Donchin (1983), who showed that 



while P300 to a secondary task decreased with in- 
crease in task difficulty, P300 to an event embedded 
within the primary task increased. Thus there is a 
reciprocal relation between P300 amplitude and re- 
source allocation between primary and secondary 
tasks in a dual-task situation. Further evidence for 
reciprocity of primary and secondary task resources 
was provided in a subsequent study by Sirevaag, 
Kramer, Coles, and Donchin (1989). This pattern of 
P300 reciprocity is consistent with resource trade- 
offs predicted by the multiple resource theory of 
Wickens (1984). 

A unique feature of the P300 is that it is simul- 
taneously sensitive to the allocation of attentional 
resources (P300 amplitude) and to the timing of 
stimulus identification and categorization processes 
(P300 latency). These features allow not only for as- 
sessing the workload associated with dual- or mul- 
titask performance, but also allow for identifying 
the sources that contribute to workload and dual- 
task interference. These features were elegantly ex- 
ploited in a dual-task study by Luck (1998) using 
the psychological refractory period (PRP) task 
(Pashler, 1994). In the PRP paradigm, two targets 
are presented in succession and the participant 
must respond to both. The typical finding is that 
when the interval between the two targets is short, 
RT to the second target is substantially delayed. 
Luck (1998) identified the source of this interfer- 
ence by recording the P300 to the second target 
stimulus, which was either the letter X or the letter 
O, with one of the letters being presented only 25% 
of the time (the oddball). Luck (1998) found that 
whereas RT was significantly increased when the in- 
terval between the two stimuli was short, P300 la- 
tency was only slightly increased. This suggested 
that the primary source of dual-task interference 
was the response selection stage, which affected RT 
but not P300 latency. 

Despite the large body of evidence confirming 
the sensitivity of P300 to perceptual and cognitive 
workload, and the smaller body of work on the 
sensitivity of Nl (or Nd), the issue of whether 
P300 can be used to track dynamic variation in 
workload, in real time, or in near-real time, has not 
been fully resolved. On the one hand, P300 is a rel- 
atively large ERP component, so that its measure- 
ment on single (or a few) trials is easier than for 
other ERP components. However, the question re- 
mains whether P300 amplitude computed on the 
basis of just a few trials can reliably discriminate 
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between different levels of cognitive workload. 
Humphrey and Kramer (1994) provided impor- 
tant information relevant to this issue. They had 
participants perform two complex, continuous 
tasks, gauge-monitoring and mental arithmetic, and 
recorded ERPs to discrete events from each task. 
The difficulty of each was manipulated to create 
low- and high-workload conditions. The amplitude 
of the ERP (averaged over many samples) was sensi- 
tive to increased processing demands in each task. 
Humphrey and Kramer then used a stepwise dis- 
criminant analysis to ascertain how the number of 
ERP samples underlying P300 affected accuracy 
of classification of the low- and high-workload con- 
ditions. They found that classification accuracy in- 
creased monotonically with the number of ERP 
samples, but that high accuracy (approximately 
90%) could be achieved with relatively few samples 
(5-10). Such a small number of samples can be col- 
lected relatively quickly, say in about 25-50 sec- 
onds, assuming a 20% oddball target frequency and 
a 1-second interstimulus interval. These results are 
encouraging with respect to the use of single-trial 
P300 for dynamic assessment of cognitive workload. 

The studies conducted to date indicate that the 
sensitivity of ERPs to temporal aspects of neural ac- 
tivation has been put to good use in dissecting 
sources of dual-task interference and in assessing 
mental workload. Furthermore, as illustrated by 
the research of Humphrey and Kramer (1994), 
there has also been some success in using ERP 
components as near real-time measures of mental 
workload. Furthermore, flight and driving simula- 
tion studies have shown the added value that ERPs 
can provide in assessment of workload in complex 
tasks. However, additional work is needed with 
single-trial ERPs to further validate the use of ERPs 
for real-time workload assessment. 

Further progress in this area may come sooner 
rather than later because of developments in the 
related field of brain-computer interface (BCI). 
This refers to the use of brain signals to control 
external devices without the need for motor out- 
put. Such brain-based control would be advanta- 
geous for individuals who either have only limited 
or no motor control, such as "locked-in" patients 
with amyotrophic lateral sclerosis. The idea of BCIs 
follows naturally from the work on biocybernetics 
in the 1980s pioneered by Donchin and others 
(Donchin, 1980; Gomer, 1981). The goal of biocy- 
bernetics was to use EEG and ERPs as an additional 



communication channel in human-machine inter- 
action. Similarly, BCI researchers hope to provide 
those with limited communication abilities addi- 
tional means of communicating and interacting with 
the world. BCIs require that brain signals for control 
be analyzed in real time within a short period of 
time, so as to give the patient adequate control of an 
external device. Hence, many BCI researchers are 
examining various EEG and ERP components that 
can be detected in single or very few trials. These in- 
clude EEG mu rhythms (Pfurtscheller & Neuper, 
2001), P300 (Donchin, Spence, & Wijesinghe, 
2000), steady-state evoked potentials (McFarland, 
Sarnacki, & Wolpaw, 2003), and contingent nega- 
tive variation and other slow potentials (Birnbaumer 
et al, 1999). The outcome of this program of re- 
search and development is likely to have a great in- 
fluence on the development of real-time ERP and 
EEG-based systems for aiding nondisabled persons 
(see also chapter 20, this volume). 

Attentional Resources and the PI 
and Nl Components 

Although most ERP research on mental workload 
has focused on the P300 component, other ERP 
components have also been examined in a few 
studies. Attentional resource allocation can also be 
manifested in the amplitude of ERP components 
earlier than P300, such as Nl. Nl is an early com- 
ponent of ERPs with a peak latency of about 100 ms 
in audition and about 160 ms in vision (Naatanen, 
1992). It has been proposed that the Nl compo- 
nent provides an index of resource allocation un- 
der high information load conditions (Hink, Van 
Voorhis, Hillyard, & Smith, 1977; Parasuraman, 
1978, 1985). 

Parasuraman (1985) had participants perform 
a visual and an auditory discrimination task con- 
currently, and systematically varied the priority to 
be placed on one task relative to the other, from 
0% to 100%. The effect of information load on 
attention allocation was also investigated by ma- 
nipulating the presentation rates (slow vs. fast). 
Performance operating characteristics showed that 
the priority instructions were successful, in that 
performance on one task varied directly with task 
priority to that task, whereas performance on the 
other task varied inversely with priority. The am- 
plitudes of the visual N160 and P250 and the audi- 
tory N100 components also varied directly and in a 
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graded manner with task priority. However, these 
graded changes in amplitude for visual or auditory 
ERP components occurred only when the stimulus 
presentation rate was high, suggesting that a cer- 
tain amount of workload is necessary for resource 
allocation between the two channels. These results 
complement those described earlier for P300 and 
show that these early latency components also ex- 
hibit resource reciprocity. 

The auditory N100 component is known to 
consist of both an exogenous N100 component 
and an endogenous Nd component that is modu- 
lated by attention (Naatanen, 1992). It is therefore 
of interest to note that the Nd component is also 
resource sensitive. Singhal, Doerfling, and Fowler 
(2002) had participants perform a dichotic listen- 
ing task while engaged in a simulated flight task of 
varying levels of difficulty. The amplitudes of both 
Nd and P300 were reduced at the highest level of 
difficulty. 

Visual selective attention can also modulate the 
amplitude of the early PI component (70-110 ms; 
Fu, Caggiano, Greenwood, & Parasuraman, 2005; 
Fu, Fan, Chen, & Zhuo, 2001; Hillyard & Munte, 
1984; Mangun, 1995). This early Pl/Nl atten- 
tional modulation contributes to the debate on 
early versus late selection in cognitive psychology 
by providing evidence that attention selects visual 
information at the early processing stage, rather 
than at later a selection stage such as decision 
making or response. For example, in a sustained 
attention study by Hillyard and Munte (1984), par- 
ticipants were asked to selectively attend to the left 
or right visual field and ignore the other visual field 
during separate blocks while maintaining their 
gaze on the central fixation point. It was found that 
the attended stimulus elicited larger PI and Nl rel- 
ative to the same stimulus at the same visual field 
when it was ignored or unattended (i.e., partici- 
pants attended to the stimulus in the opposite 
visual field). This Pl/Nl enhancement by visual se- 
lective attention is considered to reflect an early 
"sensory gating" mechanism of visual-spatial atten- 
tion (Hillyard & Anllo-Vento, 1998), and has been 
replicated across a variety of tasks, such as sus- 
tained attention and trial-by-trial cueing tasks 
(Mangun, 1995) and visual search tasks (Luck, Fan, 
& Hillyard, 1993). 

Some ERP evidence suggests that the earliest 
attentional modulation occurs in the extrastriate 
cortex (the source of the PI attentional effect), but 



not in the striate cortex (the source for the CI com- 
ponent; Clark et al., 1995; Clark & Hillyard, 1996). 
A feedback mechanism has been proposed to ac- 
count for the role of striate cortex in the visual pro- 
cessing, which suggests an anatomically early but 
temporally later striate cortex activity after the 
reentrant process from higher visual cortex onto 
VI (Martinez et al., 1999, 2001; Mehta, Ulbert, & 
Schroeder, 2000a, 2000b). By such feedback or 
reentrant processes, neural refractoriness may be 
reduced and the perceptual salience of attended 
stimuli may be enhanced (Mehta et al., 2000a, 
2000b). Alternatively, the figure-ground contrast 
and the salience of attended stimuli may be en- 
hanced (Lamme & Spekreijse, 2000; Super, Spekrei- 
jse, & Lamm, 2001). This is consistent with a recent 
finding of striate cortex activation by attention in 
the brain imaging literature (Gandhi, Heeger, & 
Boynton, 1999; Somers, Dale, Seiffert, & Tootell, 
1999; Tootell et al., 1998). However, whether the 
initial processing in the striate cortex (indexed by 
the CI component) is modulated by attention is 
still controversial. It is possible that the previous 
studies might have not adopted the optimal experi- 
mental conditions for investigating the role of stri- 
ate cortex in early visual processing (Fu, Caggiano, 
Greenwood, & Parasuraman, 2005; Fu, Huang, 
Luo, Greenwood, & Parasuraman, 2004). 

While the results of these studies on the locus 
of attentional selection in the brain are primarily 
relevant to basic theoretical issues in cognitive neu- 
roscience, there are implications for neuroergonom- 
ics as well. First, the results indicate that several 
more ERP candidate components, such as CI and 
Nl, in addition to the later P300, might be sensi- 
tive to the effects of workload or stimulus load on 
the operator and may contribute to successful eval- 
uation of operator mental state. Second, the PI 
component is a very sensitive index of the alloca- 
tion of visuospatial attention. Consequently, it 
would be very informative to monitor the opera- 
tor's attentional allocation in tasks involving spatial 
selection, such as when a driver is driving (see fig- 
ure 3.2 for attentional modulation of the PI com- 
ponent). Finally, the earlier ERP components such 
as CI, PI, and Nl usually have a relatively well- 
understood psychological meaning and involve 
less complex brain mechanisms than the later ERP 
components such as P300; in terms of source local- 
ization, these earlier components can be also local- 
ized more easily and with less error. Thus these 
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earlier ERP components can provide an assessment 
of neural mechanisms in a more accurate way un- 
der conditions when a detailed evaluation of the 
operator's mental status is needed. However, to 
what extent these ERP components can be reliably 
used in complex tasks remains to be seen. One po- 
tential limitation is the requirement of a higher 
number of trials for EEG averaging as compared to 
the later and larger components such as P300. 
Thus the use of early latency ERP components may 
require longer recording times and may be less effi- 
cient in real-time monitoring of a human operator's 
mental state as compared with the P300. 

Error Detection and Performance 
Monitoring 

Another important area for neuroergonomic re- 
search using ERPs is the analysis and possible pre- 
diction of human error. Cognitive scientists and 
human factors analysts have proposed many differ- 
ent approaches to the classification, description, and 
explanation of human error in complex human- 
machine systems (Norman & Shallice, 1986; Rea- 
son, 1990; Senders & Moray, 1991). Analysis of 
brain activity associated with errors can help refine 
these taxonomies, particularly in leading to testable 
hypotheses concerning the elicitation of error. The 
neural signature of an error may provide important 
information in this regard. 

One such neural sign is a specific ERP com- 
ponent associated with error, error-related negativ- 
ity or ERN. The ERN, which has a frontocentral 
distribution over the scalp, reaches a peak about 
100-150 ms after the onset of the erroneous re- 
sponse (as revealed by measures of electromyo- 
graphic activity) and is smaller or absent following 
a correct response (Gerhring, Goss, Coles, Meyer, 
& Donchin, 1993). The ERN amplitude is related 
to perceived accuracy, or the extent to which par- 
ticipants are aware of their errors (Scheffers & 
Coles, 2000). Importantly, the ERN seems to reflect 
central mechanisms and is relatively independent of 
output modality. Holroyd, Dien, and Coles (1998) 
found that errors made in a choice reaction time 
task in which either the hand or the foot was used 
to respond led to nearly identical ERN. 

The ERN was first identified in studies in which 
ERPs were selectively averaged to correct and incor- 
rect responses in discrimination tasks (Gehring 
et al., 1993). In choice RT tasks, a negative-going 



potential is observed at anterior recording sites on 
trials in which participants make errors. The am- 
plitude of this potential was found to be larger 
when the task instruction emphasized response ac- 
curacy over response speed — hence the label error- 
related negativity (Gehring et al., 1993). The ERN 
appears to be a manifestation of this error signal, 
with its amplitude reflecting the degree of mis- 
match between the two representations of the error 
response and the correct response, or the degree 
of error detected by the participants. That is, the 
greater the difference between the error response 
and the correct response, the larger the ERN ampli- 
tude; or the more the participants realize their er- 
rors, the larger the ERP amplitude. 

The ERN may also be considered to be a neural 
response related to performance monitoring activi- 
ties, or a comparison between representations of 
the appropriate response and the response actually 
made (Bernstein, Scheffers, & Coles, 1995; Schef- 
fers & Coles, 2000). The amplitude of the ERN 
therefore could provide a useful index of the per- 
ceived inaccuracy of the operator's performance, 
given that Scheffers and Coles showed that larger 
ERNs are associated with errors due to premature 
responding (known errors), whereas smaller ERNs 
are associated with errors due to data limitations 
(uncertain errors). Scheffers and Coles used the 
Eriksen flanker task, in which a central letter in a 
five-letter array was designated as the target for de- 
tection (H or S). The flanker stimuli could be com- 
patible (HHHHH and SSSSS) or incompatible 
(HHSHH, SSHSS) with the central target letter. 
Following each target detection response, partici- 
pants were asked to rate their confidence using a 5- 
point scale of sure correct, unsure correct, don't know, 
unsure incorrect, or sure incorrect. Response-locked 
ERPs were separately averaged for trials with cor- 
rect and incorrect responses. The typical ERN 
was observed, with its amplitude being larger for 
incorrect than for correct trials at frontal scalp sites. 
Furthermore, on incorrect trials, the greater the 
confidence participants had that their responses 
were wrong, the larger the ERN amplitude. For 
correct trials, the less that participants felt confi- 
dent about their response (i.e., rating their correct 
response from sure correct to sure incorrect), the 
larger the ERN amplitude. Therefore, there was a 
systematic relationship between the participants' 
subjective perception of their response accuracy and 
ERN amplitude, with the smallest ERN associated 
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with perceived-correct responses and increasingly 
larger ERN associated with perceived-incorrect re- 
sponses. These results were consistent with the idea 
that the ERN is associated with the detection of er- 
rors during task performance. 

Scheffers and Coles (2000) also investigated 
the relationship between type of errors and ERN 
amplitude. When participants judged their incor- 
rect response as sure incorrect, they must have had 
enough information to be aware of their wrong re- 
sponse, indicating that this type of error was due 
to a premature response. On the other hand, re- 
sponses judged as don't know were probably due to 
data limitations associated with insufficient stimu- 
lus quality or by the presence of incompatible 
flanking stimuli. Assuming that ERN is elicited by 
the comparison between the appropriate and the 
actual response, premature responses should be as- 
sociated with this comparison and therefore should 
elicit larger ERN. On the other hand, data limita- 
tions should be associated with a compromised 
representation of the appropriate response, so that 
only a partial mismatch is involved, and therefore 
should elicit smaller ERN. Their results confirmed 
that errors due to premature responses elicited 
larger ERN, whereas those due to data limitations 
were associated with ERN of intermediate ampli- 
tude, supporting the view that ERN is a manifesta- 
tion of a process that monitors the accuracy of 
performance. 

The ERN may be generated in the anterior cin- 
gulate cortex (ACC) in the frontal lobe (Dehaene, 
Posner, & Tucker, 1994; Miltner, Braun, & Coles, 
1997). As such, the ERN might reflect monitoring 
processes at a lower level of control rather than be- 
ing associated with the detection and evaluation of 
errors at the higher level of supervisory executive 
control that is mediated by the prefrontal cortex. 
Moreover, ERN, and the ACC activation it repre- 
sents, might be a manifestation of a monitoring 
process that specifically detects errors rather than 
conflict in general (Scheffers & Coles, 2000). 

Rollnik et al. (2004) combined ERP and repeti- 
tive transcranial magnetic stimulation (rTMS) tech- 
niques to investigate the role of medial frontal 
cortex (including ACC) and dorsolateral prefrontal 
cortex (DLPFC) in performance monitoring. They 
found that ERN errors in an Eriksen flanker task 
elicited the ERN. Moreover, rTMS of the medial 
frontal cortex attenuated ERN amplitude and in- 
creased the subsequent error positivity (Pe) relative 



to a no-stimulation (control) condition, whereas no 
such effect was observed after lateral frontal stimu- 
lation, suggesting that the medial frontal cortex is 
important for error detection and correction. 

The relevance of ERN to neuroergonomic re- 
search and applications is straightforward. The 
ERN allows identification, prediction, and perhaps 
prevention of operator errors in real time. For ex- 
ample, the ERN could be used to identify the hu- 
man operator tendency to either commit, recognize, 
or correct an error. This could potentially be de- 
tected covertly by online measurement of ERN 
prior to the actual occurrence of the error, given 
that the ERN could be reliably measured on a sin- 
gle trial. Theoretically a system could be activated 
by an ERN detector in order to either take control 
of the situation (e.g., in those cases where time to 
act is an issue) or to notify the operator about the 
error he or she committed, even providing an adap- 
tive interface that selectively presents the critical 
subsystems or function. Such a system would have 
the advantage of keeping the operator still in con- 
trol of the entire system, while providing an anchor 
for troubleshooting when the error actually occurs 
(and having the possibility, for the system, to cor- 
rect it by itself if needed). 

Response Readiness 

It has long been known that scalp ERPs can be 
recorded prior to the onset of a motor action. Such 
potentials are known as readiness potentials (Bereit- 
schaftpotential; Kornhuber & Deecke, 1965). The 
lateralized readiness potential (LRP) is derived 
from the readiness potentials that occur several 
hundred milliseconds before a hand movement 
(Gratton, Coles, Sirevaag, Eriksen, & Donchin, 
1988). The LRP is normally more pronounced on 
the contralateral scalp site of the responding hand, 
relative to the ipsilateral site. Typically, scalp elec- 
trodes at the C3 and C4 sites overlying motor cor- 
tex are used to record the LRP These characteristics 
suggest that the LRP might be a good index of the 
covert process of movement preparation (Kutas & 
Donchin, 1980). The asymmetry between contra- 
and ipsilateral readiness potential specific to hand 
movement is defined by LRP, which is the average 
of the two difference waves obtained by subtracting 
the readiness potential from the ipsilateral to that 
of the contralateral hemisphere for left- and right- 
hand responses, respectively (Coles, 1989; Gratton 
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et al., 1988). The averaging procedure is applied to 
remove all the non-movement-related asymmetric 
effects because they remain constant when the side 
of movement changes. The resulting LRP is thought 
to reflect the time course of the activation of the 
left and right motor responses; that is, the LRP can 
indicate whether and when a motor response is se- 
lected. It has been further proposed that the inter- 
val between the stimulus onset and LRP onset 
indicates the processes prior to the response hand 
selection, whereas the interval between the onset of 
LRP and the completion of the motor response re- 
flects response organization and execution pro- 
cesses (Osman & Moore, 1993). 

The LRP has been combined with other ERP 
components such as ERN to obtain converging evi- 
dence of response selection and execution processes 
in relation to error. For example, Van Schie, Mars, 
Coles, and Bekkering (2004) analyzed both the 
ERN and the LRP to investigate the neural mecha- 
nisms underlying error processing and showed that 
activity in both the medial frontal cortex (as mea- 
sured by ERN) and the motor cortices (as mea- 
sured by LRP) was modulated by the correctness of 
both self-executed and observed action. 

Even though modern human-machine systems 
are becoming increasingly automated (Parasura- 
man & Mouloua, 1996), so that human operator 
actions are limited, many tasks of everyday life 
continue to require significant amounts of motor 
output. To the extent that such motor actions con- 
tribute to system performance and safety motor- 
related brain potentials such as the LRP can be 
used to assess the speed efficiency of such actions 
and their underlying neural basis. Thus, the LRP is 
another useful tool for neuroergonomics in systems 
such as driving or keyboard entry, where motor ac- 
tions are a key feature. 

Assessment of Automatic Processing 

One other potential ERP index that can be of use to 
neuroergonomics is the MMN. The MMN, which 
was discovered by Naatanen and colleagues, is 
considered to reflect automatic processing of audi- 
tory information (for reviews, see Naatanen, 1990; 
Naatanen & Winkler, 1999). The typical paradigm 
used to elicit the MMN is the oddball paradigm, in 
which an infrequent deviant auditory stimulus 
(tone, click, etc.) is embedded in a sequence of 
repeated standard stimuli. The difference wave 



obtained by subtracting the ERPs to deviant stimuli 
from the standard stimuli produces the MMN. The 
MMN has an onset latency of 100 ms and has 
maximal distribution over the frontal area. Recent 
neuroimaging studies have confirmed that the 
source of the MMN can be localized to the bilateral 
supratemporal cortex and the right frontal cortex 
(Naatanen, 2001). 

The MMN might reflect, at least partially, auto- 
matic auditory change detection between the occa- 
sional deviant stimuli and the memory trace 
developed by the repetition of the standard stimuli, 
because it shows a similar amplitude when subjects 
attend to stimuli and when they ignore them, and 
it can be obtained when participants' attention is 
directed to another modality, such as vision. But 
the MMN might not completely reflect automatic 
processing, because its amplitude under certain 
conditions can vary with the load in the attended 
modality (Woldorff, Hackley, & Hillyard, 1991). 

Several characteristics of the MMN might be 
useful for neuroergonomics. First, there is a corre- 
lation between MMN latency and RT, because both 
diminish when the physical difference between the 
deviant and standard stimuli is increased. There- 
fore, the MMN latency might be used as an index 
of response speed when no overt responses are 
available from the operator. The other is that the 
latency of MMN is inversely related to, and the am- 
plitude of MMN is positively related to, the magni- 
tude of the difference between the deviants and the 
standards. Furthermore, the discrimination process 
generating MMN might be important to involun- 
tary orienting or attentional switching to a change 
in the acoustic environment. 

Whether MMN is specific to the auditory 
modality is controversial (for a recent review, see 
Pazo-Alvarez, Cadaveira, & Amenedo, 2003). Some 
recent studies indicate that there might be a visual 
MMN that is an analog of the auditory MMN 
(Czigler & Csibra, 1990, 1992; Fu, Fan, & Chen, 
2003). However, the studies for visual MMN should 
include the necessary experimental controls to af- 
firm that there is a reliable homologue of the audi- 
tory MMN, by illustrating its underlying neural 
sources, the optimal parameters and property devi- 
ations to elicit it, and its characteristics as compared 
with auditory MMN. Confirmation of a visual MMN 
would be of considerable interest, because of the 
greater range of potential applications to visual pro- 
cessing issues in human factors research. 
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Conclusion 

We have discussed the contribution of ERPs to hu- 
man factors research and practice by describing the 
specific methodological advantages that ERPs pro- 
vide over other measures of human brain function. 
ERPs have provided some specific insights not avail- 
able from other measures in such areas as mental 
workload assessment, attentional resource alloca- 
tion, error detection, and brain-computer interfaces. 
In a previous review, Parasuraman (1990) con- 
cluded that while the role of ERPs in human factors 
is modest, that role in a small number of cases is 
highly significant. Years later, with the prosperity of 
cognitive neuroscience and its penetration to fields 
such as cognitive psychology, the high-temporal- 
resolution and moderate-spatial-resolution ERP 
technique may be of greater importance now with 
the development of neuroergonomics. At the same 
time, it must be acknowledged that neuroergonomic 
applications of ERPs are still small in number. With 
further technical developments in miniaturization 
and portability of ERP systems, such practical appli- 
cations may prosper. Given the continued devel- 
opment of automated systems in which human 
operators monitor rather than actively control sys- 
tem functions, there will be numerous additional 
opportunities for the use of ERP-based neuroer- 
gonomic measures and theories to provide insights 
into the role of brain function at work. 



Main Points 

1 . ERPs represent the brain's neural response to 
specific sensory, motor, and cognitive events 
and are computed by averaging EEG epochs 
time-locked to a particular event. 

2. Compared to other neuroimaging techniques, 
ERPs provide the best temporal resolution 

(1 ms or better) for evaluating neural activity 
in the human brain. 

3. The spatial resolution of neuronal activity 
provided by ERPs is poor compared to fMRl, 
but can be improved through the use of 
various source localization techniques. 

4. ERP components such as P300, Nl, PI, ERN, 
and LRP can provide information on the neural 
basis of functions critical to a number of human 
factors issues. These include the assessment of 



mental workload, attention resource allocation, 
dual-task performance, error detection and 
prediction, and motor control. 
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Functional Magnetic Resonance 
Imaging (fMRI) 

Advanced Methods and Applications 
to Driving 



Functional magnetic resonance imaging (fMRI) is 
now able to provide an unprecedented view inside 
the living human brain. Early studies focused 
upon carefully designed, but often artificial, para- 
digms for probing specific neural systems and 
their modulation (Cabeza & Nyberg, 2000). Such 
approaches, though important, do not speak di- 
rectly to how the brain is performing a complex 
real-world task. Relatively real-world or naturalis- 
tic tasks often involve multiple cognitive domains 
and may well result in emergent properties not 
present in tasks that probe separate cognitive do- 
mains. Performing a naturalistic task is obviously 
quite complex, and the difficulty of analyzing and 
interpreting the data produced by these paradigms 
has hindered the performance of brain imaging 
studies in this area. However, there is growing in- 
terest in studying complex tasks using fMRI, and 
new analytic techniques are enabling this exciting 
possibility. In this chapter, I focus upon the collec- 
tion and analysis of fMRI data during the perfor- 
mance of naturalistic tasks. Specifically, I use the 
exemplar of simulated driving to demonstrate the 
development of a brain activation model for simu- 
lated driving. 



Essentials of fMRI 

fMRI is a brain imaging technique in which relative 
changes in cerebral oxygenation can be assessed 
nonivasively while a person is engaged in perform- 
ing cognitive tasks. Participants lie supine in the 
MRI scanner while exposed to a static magnetic 
field, typically 1.5 or 3 Teslas (1.5 or 3 T). A radio 
frequency coil placed near the head is then briefly 
pulsed to temporarily disturb the aligned magnetic 
field passing through the brain, and the resulting 
magnetic resonance signal that is emitted as the 
field settles back into its static state is detected. 
fMRI exploits the fact that oxygenated blood has 
different magnetic properties than deoxygenated 
blood or the surrounding tissues, as a result of 
which the strength of the magnetic resonance sig- 
nal varies with the level of blood oxygenation. As a 
given brain region increases its neural activity, 
there is initially a small depletion in the local oxy- 
genated blood pool. A short while later, however, 
the cerebrovasculature responds to this oxygen 
depletion by increasing the flow of oxygenated 
blood into the region for a short time, before re- 
turning oxygenated blood levels back to normal. 
For reasons that are not completely understood, 
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the supply of oxygenated blood exceeds the neural 
demand, so that the ratio of oxygenated to de- 
oxygenated blood is altered. It is this disparity — the 
blood oxygen level-dependent (BOLD) signal — that 
is measured and tracked throughout the brain. Re- 
searchers can then use the BOLD signal and associ- 
ate it with performance of a cognitive task, in 
comparison to a baseline or resting state, or to an- 
other cognitive task that differs in requiring an ad- 
ditional cognitive operation. Subtraction is used to 
isolate the activation patterns associated with the 
cognitive operation. This is the basic block design 
of many fMRI studies and allows the researcher to 
identify cortical regions that are involved with per- 
ceptual, cognitive, emotional, and motor operations 
(see Cabeza & Nyberg, 2000; Frackowiak et al., 
2003, for reviews). This chapter focuses on more 
sophisticated fMRI analytical techniques to assess 
changes in brain activity in persons engaged in a 
complex activity of daily living, driving. 



Why Study Simulated Driving 
With fMRI? 

Driving is a complex behavior involving inter- 
related cognitive elements including selected and 
divided attention, visuospatial interpretation, visuo- 
motor integration, and decision making. Several 
cognitive models have been proposed for driving 
(Ranney, 1994), especially for visual processing as- 
pects and driver attributes (Ballard, Hayhoe, Sal- 
gian, & Shinoda, 2000; Groeger, 2000). However, 
such models are complicated and hard to translate 
into imaging studies. Many of the cognitive ele- 
ments expected to be involved in driving have been 
studied separately using imaging paradigms de- 
signed to probe discrete brain systems (Cabeza & 
Nyberg, 2000). Imaging studies have used subtrac- 
tive methods to study the neural correlates of driv- 
ing (Walter et al., 2001) but have not attempted to 
study the complex temporal dynamics of driving. 



brain the MRI time course resembled the para- 
digm time courses (e.g., the task-baseline cycle in 
the block design is convolved with the hemody- 
namic impulse response). The data are then ana- 
lyzed using a variety of statistical techniques, from 
simple t tests and analysis of variance (ANOVA) to 
the general linear model (GLM; Worsley & Fris- 
ton, 1995). The GLM method is a massively uni- 
variate inferential approach in which the same 
hypothesis (the brain time course resembles the 
paradigm time course) is tested repeatedly at every 
location (voxel) in the brain. Many fMRI studies 
use the block design, in which cognitive tasks typ- 
ically employ subtraction between two types of 
tasks modified in slight increments (Cabeza & Ny- 
berg, 2000) and provide visualization of brain re- 
gions that differ. 

While much basic knowledge has been gained 
from the block design method, it suffers from the 
limitation that there is no attempt to study the tem- 
poral dynamics of brain activation in response to 
a cognitive challenge. Event-related fMRI, in which 
MRI signals are correlated with the timing of indi- 
vidual stimulus and response events, can be used to 
examine temporal dynamics (Friston et al., 1998). 
An advantage of using an event-related design to ex- 
amine fMRI data collected during simulated driving 
would be the ability to assess the times of unique 
discrete events such as car crashes. As most com- 
monly implemented, however, event-related fMRI 
typically requires fairly stringent assumptions about 
the hemodynamic response resulting from such an 
event (Friston et al., 1998). More flexible modeling 
can be done by estimating the hemodynamic im- 
pulse response through deconvolution approaches 
(Glover, 1999). However, all of these approaches re- 
quire a well-defined set of stimuli. For a simulated 
driving task, the stimuli (e.g., many types of visual 
stimuli, motor responses, crashes, driving off the 
road, etc.) are overlapping and continuous, and it is 
difficult to generate a well-defined set of stimuli. 



Challenges in fMRI Data Analysis 

The BOLD technique yields signals that are sensi- 
tized to hemodynamic sequelae of brain activity. 
These time-dependent data are typically reduced, 
using knowledge of the timing of events in the task 
paradigm, to graphical depictions of where in the 



Data-Driven Approach 

In addition to univariate model-based approaches, 
a variety of multivariate data-driven approaches 
have been used to reveal patterns in fMRI data. 
Among the data-driven approaches applied to fMRI 
data, we and others (Biswal & Ulmer, 1999; Cal- 
houn, Adali, Pearlson, & Pekar, 2001c; Calhoun, 
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Figure 4.1 . Comparison of general linear model (GLM) and independent component analysis (ICA; left) and ICA 
illustration (right). The GLM (top left) is by far the most common approach to analyzing fMRI data, and to use this 
approach, one needs a model for the fMRI time course, whereas in spatial ICA (bottom left), there is no explicit 
temporal model for the fMRI time course. This is estimated along with the hemodynamic source locations (right). 
The ICA model assumes the fMRI data, x, is a linear mixture of statistically independent sources s, and the goal of 
ICA is to separate the sources given the mixed data and thus determine the s and A matrices. See also color insert. 



Adali, Hansen, Larsen, & Pekar, 2003; McKeown, 
Hansen, & Sejnowski, 2003; McKeown et al., 
1998) have shown that spatial independent compo- 
nent analysis (ICA) is especially useful at revealing 
brain activity not anticipated by the investigator, 
and at separating brain activity into networks of 
brain regions with synchronous BOLD fMRI sig- 
nals. Independent component analysis was origi- 
nally developed to solve problems similar to the 
"cocktail party" problem (Bell & Sejnowski, 1995). 
The ICA algorithm, assuming independence in time 
(independence of the voices), can separate mixed 
signals into individual sources (voices). In our ap- 
plication, we assume independence of the hemody- 
namic source locations from the fMRI data 
(independence in space), resulting in maps for each 
of these regions, as well as the time course repre- 
senting the fMRI hemodynamics. 

The principle of ICA of fMRI is illustrated in 
figure 4.1. Two sources are presented along with 
their representative hemodynamic mixing func- 
tions. In the fMRI data, these sources are mixed by 
their mixing functions and added to one another. 
The goal of ICA is then to unmix the sources using 
some measure of statistical independence. The ICA 
approach does not assume a form for the temporal 
behavior of the components and thus is called a 
blind source separation method. Semiblind ICA 



methods, which have been applied to fMRI data, 
can be used to provide some of the advantages of 
both the general linear model and ICA, although 
they have not yet been applied to fMRI of simulated 
driving studies (Calhoun, Adali, Stevens, Kiehl, & 
Pekar, 2005). 

The ICA approach has also been extended to al- 
low for the analysis of multiple subjects (Calhoun, 
Adali, Pearlson, & Pekar, 2001a; Woods, 1996). We 
applied this method to our driving activation data, 
analyzing the data from all subjects in a single ICA 
estimation. This provides a way to extract behav- 
ioral correlates without having an a priori hemody- 
namic model. It also enables us to estimate the 
significance of the voxels in each component by 
computing the variance across individuals. 



The Driving Paradigm 

The driving scenario consisted of three repeating 
conditions, resembling a standard block design. 
While these three conditions provided a way to com- 
pare behavior, we did not rely upon simple compari- 
son of the images between different conditions, as in 
most fMRI block designs, but rather examined the 
source locations and the modulation of the temporal 
dynamics. This approach thus provided a useful way 
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of analyzing complex behaviors not possible using 
traditional (between-epoch) fMRI comparisons. Im- 
portantly, the activity represented by several of these 
components (e.g., anterior cingulate and frontopari- 
etal) was not found when we analyzed our data us- 
ing the GLM, or in a similar fMRI of driving study 
using only GLM analysis (Walter et al, 2001). This 
suggests that ICA may be especially useful for analy- 
sis of fMRI data acquired during such rich naturalis- 
tic behavior, as opposed to the experimental 
paradigms typical of the fMRI literature where GLM 
analysis is so prevalent. 

We also investigated whether driving speed 
would modulate the associated neural correlates 
differently. Driving at a faster speed was not ex- 
pected to modulate primary visual regions much 
since the visual objects seen are no more compli- 
cated than for the slower speed. Likewise, a speed 
change was expected to make little difference in 
the rate of finger movement or motor activation. 
However, increasing speed may increase activation 
in regions subserving eye movements or visual at- 
tention (Hopfinger, Buonocore, & Mangun, 2000) 
or anxiety. We thus scanned two groups of subjects 
driving at different rates of speed. 



Experiments and Methods 

Participants 

Subjects (2 female, 10 male; mean age 22.5 years) 
were approved by the institutional review board 
and were compensated for their participation. Sub- 
jects were given additional compensation if they 
stayed within a prespecified speed range. Subjects 
were screened with a complete physical and neuro- 
logical examination, and urine toxicological testing, 
as well as the SCAN interview (Janca, Ustun, & Sar- 
torius, 1994), to eliminate participants with Axis I 
psychiatric disorders. Subjects were divided into 
two groups (three subjects were in both groups), 
one group driving faster (N = 8) and the other 
group driving slower (N= 7). Three subjects were 
included in both groups and thus scanned during 
both speeds. The slower and faster driving groups 
were delineated by changing the speedometer dis- 
play units from kilometers per hour (kph) to miles 
per hour (mph), resulting in an actual speed range 
of 100-140 or 160-224 kph, respectively. 



Experimental Design 

We obtained fMRI scans of subjects as they twice 
performed a 10-minute task consisting of 1 -minute 
epochs of (a) an asterisk fixation task, (b) active 
simulated driving, and (c) watching a simulated 
driving scene (while randomly moving fingers over 
the controller). Epochs (b) and (c) were switched 
in the second run, and the order was counterbal- 
anced across subjects. During the driving epoch, 
participants were performing simulated driving us- 
ing a modified game pad controller with buttons 
for left, right, acceleration, and braking. The para- 
digm is illustrated in figure 4.2. 

The simulator used was a commercially avail- 
able driving game, Need for Speed II (Electronic 
Arts, 1998). Simulators of this type are both inex- 
pensive and relatively realistic. We have previously 
demonstrated the validity of a similar simulated 
driving environment compared directly to real on- 
road driving (McGinty, Shih, Garrett, Calhoun, & 
Pearlson, 2001). The controller was shielded in 
copper foil and connected to a computer outside 
the scanner room though a waveguide in the wall. 
All ferromagnetic screws were removed and re- 
placed by plastic components. An LCD projector 
outside the scanner room and behind the scanner 
projected through another waveguide to a translu- 
cent screen, which the subjects saw via a mirror, at- 
tached to the head coil of the fMRI scanner. The 
screen subtended approximately 25 degrees of vi- 
sual field. The watching epoch was the same for all 
subjects (a playback of a previously recorded driv- 
ing session). For the driving epoch, subjects started 
at the same point on the track with identical condi- 
tions (e.g., car type, track, traffic conditions). They 
were instructed to stay in the right lane except in 
order to pass, to avoid collisions, to stay within a 
speed range of 100-140 (the units were not speci- 
fied) and to drive normally. 

Image Acquisition 

Data were acquired at the FM Kirby Research 
Center for Functional Brain Imaging at Kennedy 
Krieger Institute on a Philips NT 1.5 Tesla scanner. 
A sagittal localizer scan was performed first, 
followed by a Tl -weighted anatomical scan 
(TR = 500 ms, TE = 30 ms, field of view = 24 cm, 
matrix = 256 x 256, slice thickness = 5 mm, gap = 
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Figure 4.2. fMRI simulated driving paradigm. The paradigm consisted often 1-minute epochs of 

(a) a fixation target, (b) driving the simulator, and (c) watching a simulation while randomly 
moving fingers over the controller. The paradigm was presented twice, changing the order of the 

(b) and (c) epochs and counterbalancing the first order across subjects. See also color insert. 



0.5 mm) consisting of 18 slices through the entire 
brain, including most of the cerebellum. Next, we 
acquired the functional scans consisting of an 
echo-planar scan (TR = 1 s, TE = 39 ms, field of 
view = 24 cm, matrix = 64 x 64, slice thickness = 
5 mm, gap = 0.5 mm) obtained consistently over a 
10-minute period per run for a total of 600 scans. 
Ten dummy scans were performed at the beginning 
to allow for longitudinal equilibrium, after which 
the simulated driving paradigm was begun. 

fMRI Data Analysis 

The images were first corrected for timing differ- 
ences between the slices using windowed Fourier 
interpolation to minimize the dependence upon 
which reference slice is used (Calhoun, Adali, 
Kraut, & Pearlson, 2000; Van de Moortele et al., 
1997). Next the data were imported into the statis- 
tical parametric mapping software package SPM99 
(Worsley & Friston, 1995). Data were motion cor- 
rected, spatially smoothed with an 8 x 8 x 1 1 mm 
Gaussian kernel, and spatially normalized into the 
standard space of Talairach and Tournoux (1998). 
The data were slightly subsampled to 3 x 3 x 4 mm, 



resulting in 53 x 63 x 28 voxels. For display, slices 
2-26 were presented. 

Independent Component Analysis 

Data from each subject were reduced from 600 to 
30 dimensions using principal component analysis 
(PCA; representing greater than 99% of the variance 
in the data). This prereduction (i.e., the incorpora- 
tion of an initial PCA stage for each subject) is a 
pragmatic step, reduces the amount of memory re- 
quired to perform the ICA estimation, and does not 
have a significant effect on the results, provided the 
number chosen is not too small (Calhoun, Adali, 
Pearlson, & Pekar, 2001a). To enable group averag- 
ing, data from all subjects were then concatenated 
and this aggregate data set reduced to 25 time 
points using PCA, followed by group independent 
component estimation using a neural network al- 
gorithm that attempted to minimize the mutual 
information of the network outputs (Bell & Se- 
jnowski, 1995). Time courses and spatial maps were 
then reconstructed for each subject and the random 
effects spatial maps thresholded at p < 0.00025 
(t = 4.5, dj= 14; ). The methods we used have been 
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implemented in a user-friendly software package 
available for download (http://icatb.sourceforge.net). 



Results 

For the fMRI motion correction, the translation 
corrections were less than half a voxel for any par- 
ticipant. Pitch, roll, and yaw rotations were gener- 
ally within 2.0 degrees or less. There were no 
visually apparent differences between the average 
motion parameters for the driving runs. Changes 
were also not significant as assessed by a t test on 
the sum of the squares of the parameters over time. 

GLM Results 

We performed a multiple regression analysis using 
the general linear model approach in which the 
driving and watch conditions were modeled with 
separate regressors for each participant and aggre- 
gated across subjects using a one-sample t test of the 
single subject images for each condition (Holmes & 
Friston, 1998) Results are presented in figure 4.3. 
Relative increases are in red and decreases in blue. 
Networks correlated with the drive regressor are de- 
picted on the left and networks correlated with the 



watch regressor are on the right. We found, as re- 
ported by others, that motor and cerebellar net- 
works were significantly different in the contrast 
between drive and watch (Walter et al, 2001). 

ICA Results 

ICA results are summarized in figure 4.4, with dif- 
ferent colors coding for each component. The re- 
sulting 25 time courses were sorted according to 
their correlation with the driving paradigm and 
then visually inspected for task-related or tran- 
siently task-related activity. Of the 25 components, 
only six demonstrated such relationships. The 
fMRI data comprise a linear mixture of each of the 
six depicted components. That is, if a given voxel 
had a high value for a given component image, the 
temporal pattern of the data resembled the tempo- 
ral pattern depicted for that component. Addition- 
ally, some areas consisted of more than one 
temporal pattern. For example, the pink and blue 
components overlapped heavily in the anterior cin- 
gulate and medial frontal regions. Selected Ta- 
lairach coordinates of the volume and maxima of 
each anatomical region within the maps are pre- 
sented in table 4.1. The ICA results were generally 
consistent with the GLM results; however, the 
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Figure 4.3. Results from the GLM-based fMRI analysis. See also color insert. 
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Figure 4.4. Independent component analysis results. Random effects group fMRI maps are thresholded at 
p < 0.00025 (t = 4.5, df = 14). A total of six components are presented. A green component extends on both sides 
of the parieto-occipital sulcus including portions of cuneus, precuneus, and the lingual gyrus. A yellow component 
contains mostly occipital areas. A white component contains bilateral visual association and parietal areas, and a 
component consisting of cerebellar and motor areas is depicted in red. Orbitofrontal and anterior cingulate areas 
identified are depicted in pink. Finally, a component including medial frontal, parietal, and posterior cingulate re- 
gions is depicted in blue. Group averaged time courses (right) for the fixate-drive-watch order are also depicted 
with similar colors. Standard deviation across the group of 15 scans is indicated for each time course with dotted 
lines. The epochs are averaged and presented as fixation, drive, and watch. See also color insert. 



GLM approach failed to reveal the complexity of 
the temporal dynamics present within the data set. 
The group-averaged time course for the 
fixation-drive-watch paradigm (with the standard 
deviation across the 15 subjects) for each compo- 
nent is presented on the right side of figure 4.4, with 
color use as in the spatial component maps. The 
three epoch cycles are averaged together and are 
presented as fixation, drive, and watch. Though it is 
intriguing to try to separate finer temporal correlates 
within the ICA time courses (such as the time of a 
crash), we did not have exact timing for individual 
events in the simulator used and thus limited our 
exploration to epoch-based averages. The ordering 
of the epochs did not significantly change the re- 
sults. Each of the time courses depicted was modu- 



lated by the driving paradigm. Four main patterns 
are apparent: (1) primary visual (yellow) and higher- 
order visual/cerebellar (white) areas were most ac- 
tive during driving and less active during watching 
(as compared to fixation) (W,Y); (2) cerebellar/motor 
(red) and frontoparietal (blue) areas were only in- 
creased or decreased during driving; (3) anterior 
cingulate, medial frontal, and other frontal (pink) ar- 
eas demonstrated exponential decrements during 
driving and rebounded during fixation; and (4) vi- 
sual (green) areas transiently activated when the 
driving or watching paradigms were changed. 

We also examined relationships between driv- 
ing speed and activation. We measured driving 
speed to verify that the speed was within the range 
specified (100-140 or 220-308 kph). These data 
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Table 4.1. Selected Talairach Coordinates of the Volume and Maxima of Each Anatomic Region 



Component Area 



Brodmann 



Volume (cc) 



Max t (x,y,z) 



Pink 



Blue 



Pink/blue 
(overlap) 



Red 



Yellow 



Green 



White 



L/R anterior cingulate 
L/R medial frontal 
L/R inferior frontal 
L/R middle frontal 
L/R superior frontal 

L/R sup., med. frontal 

L/R anterior cingulate 

L/R precuneus (parietal) 

L/R inf. parietal, sup. marg. , 

angular 

L/R inferior frontal 

L7R middle frontal 

L/R cingulate/post. cing. 

17R anterior cingulate 
L/R medial frontal 
L/R superior frontal 

L/R precentral 
L/R thalamus 
L/R cerebellum 

L/R lingual, cuneus 
L/R fusiform, mid. occip. 
L/R inferior occipital 

L/R cuneus 

L/R lingual 

L/R posterior cingulate 

L/R fusiform, inf. occip. 
L/R cerebellum 
L/R middle occipital 
L/R lingual, cuneus 
L/R parahippocampal 
L/R superior occipital 
L/R precuneus (parietal) 
L/R inferior temporal 
L/R superior parietal 



10,24,25,32,42 


12.6/11.1 


9,10,11,25,32 


8.2/11.3 


11,25(L),47 


5.4/2.2 


10,11,47 


2.7/2.1 


9,10,11 


1.6/3.8 


6,8,9,10 


33.4/29.6 


10,24,32,42 


5.2/2.5 


7,23,31 


1.6/3.1 


39,40 


1.4/2.0 


11(R),45(R),47 


0.8/3.4 


6,8 


0.3/1.2 


23,30,31,32 


1.2/1.7 


10,24,32,42 


4.5/2.4 


9,10,11,32 


4.1/3.8 


9,10 


.272 


4(L),6 


3.7/0.5 


— 


0.5/0.1 


— 


31.7/34.1 


17,18,190-) 


18.4/16.6 


18,19 


7.2/5.9 


17,18 


1.6/1.3 


17,18,19,23,30 


9.7/7.2 


17,18,19 


4.6/2.7 


30,31 


0.8/0.6 


8,19,20,36(R),37 


15.6/12.3 


— 


13.0/13.5 


18,29,37 


6.9/4.0 


7,18,19 


2.7/1.5 


19,36,37 


1.7/0.8 


19 


1.1/0.1 


7,19(L) 


0.9/7.7 


19,37 


0.7/0.4 


7 


0.4/0.5 



18.94(-9,-35,-6)/15. 38(3,43,-6) 
16.97(-12,35,6)/14.45(6,40,-6) 

13.20(-9,40,-15)/9.02(12 -40,-15) 
15. 33(-18,37,-10)/9. 34(12,43,-15) 
11. 60(-18,43,-ll)/9. 70(18,50, 16) 

14.68(-9,62,15)/ 15.83(3,50,2) 
11.44(-3,49,-2)/14.22(3,47,7) 
8. 72(-3,-54,30)/ll. 38(6,-54,30) 

7.91(-53,-59,35)/7. 29(53,-68,31) 

6.23(-39,23,-14)/11.89(48,26,-10) 

6. 94(-18,34,44)/7. 10(27,40,49) 

8.59(-3,-54,26)/11.88(6,-57,30) 

17. 52(-9,38,3)/15. 38(3,43,-6) 
15.79(-6,38,-6)/15.83(3,50,2) 
7.60(-9,52,-3)/8.43(12,51,20) 

5.24(-33,-13,60)/3.45(24,-ll,65) 
7. 30(-15,-20,-10)/4. 78(9,-14,5) 
11.18(-6,-59,-10)/13.86(6,-71,-22) 

15.H-9,-96,-9)/18.21(18,-93,5) 
13. 07(-9 -95, 145/11. 38(18,-98,14) 
8.91(-15,-91,-8)/10.48(21 -88,-8) 

22. 88(-9,-81,9)/23. 19(6,-78,9) 
21.98(-6,-78,4)/16.12(15,-58,-l) 
13. 06(-12,-64,8)/12. 33(9 -67,8) 

9.48(-48,-70,-9)/10. 56(42 -74,-17) 
9. 36(-33,-50,-19)/12.08(42 -74,-22) 
10.74(-33,-87, 18)77.1 1(33,-87,18) 



8.57(-27,-83,27)/7.22(24 
8.15(-24,-47,-10)/7. 16(27 
12.04(-33,-86,27)/6.09(33 
6.91(-27,-77,36)/7. 90(18 
6.26(-48,-73,-l)/6.04(48 
6.38(-18,-67,54)/6. 77(18 



-92,23) 

-44,-10) 

-86,23) 

-70,50) 

-71,-1) 

-70,54) 



Note. For each component, all the voxels above the threshold were entered into a database to provide anatomical and functional labels for the 
left (L) and right (R) hemispheres. The volume of activated voxels in each area is provided in cubic centimeters (cc). Within each area, the maximum 
t value and its coordinate is provided. 



are summarized for each subject in table 4.2. Note 
that the average speed for the faster group was 
slower than the range specified. This was because 
there were more collisions at the faster speed, thus 
reducing the average (although it was still signifi- 
cantly higher than the slower group). The increase 



in collisions indicates that subjects are making 
more errors, presumably because of a desire to 
comply with the specified speed range. 

The frontoparietal (blue) component was de- 
creased during the drive epoch only. Calculating the 
change in activation for the two speed groups and 
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Table 4.2. Average and Maximum Driving Speeds 
for the 15 Subjects (Kilometers per Hour) 







Slow 






Fast 




Subject 


Average 


Maximum 


Average 


Maximum 


1 


106 




142 


140 




248 


2 


106 




126 


125 




190 


3 


117 




138 


124 




220 


4 


113 




131 


122 




206 


5 


114 




140 


151 




196 


6 


120 




140 


137 




192 


7 


98 




141 


155 




246 


8 








159 




213 


Avg. 


111 




137 


139 




214 



Note. All subjects performed the task as instructed and the slow 
drivers all had average speeds lower than those of the fast drivers. 



comparing them with a t test revealed significantly 
(p < 0.02) greater driving-related changes when 
subjects were driving faster. This is depicted in fig- 
ure 4.5a and is consistent with an overall increase in 
vigilance while driving at the faster speed. Previous 
imaging studies have implicated similar frontal and 
parietal regions in visual awareness (Rees, 2001). 

Examination of the orbitofrontal/anterior cin- 
gulate (pink) time courses during the driving 
epoch revealed an exponentially decaying curve, 
displayed in figure 4.4. We extracted the signal 
decreases during the 180 (3 x 60) time points of 
driving s(n) and normalized so that the average of 
the asterisk epoch was 1 and the minimum of the 
driving epoch was (results were not sensitive 



to this normalization). Next we transformed the 
orbitofrontal time course to a linear function 
through the equation an = ln[s(n) + 1] and fit a line 
to it such that x(n) = ay (n) + b. The resultant fit 
was very good (average R = 0.9). 

Comparing the subjects driving slower with 
those driving faster using a t test revealed a signifi- 
cant difference (p < .02) in the rate parameter a. 
That is, the rait of decrease in anterior cingulate 
pink signal is faster in the subjects driving at a faster 
rate (a more difficult task). This speed-related rate 
change was also consistent with the involvement of 
the orbitofrontal and anterior cingulate involve- 
ment in disinhibition (i.e., "taking off the brake"; 
Blumer & Benson, 1975; Rauch et al., 1994). A 
comparison of the rates is depicted in figure 4.5b. 
No significant correlations were found for any of 
the other revealed components. 

The components identified by our analysis lend 
themselves naturally to interpretation in terms of 
well-known neurophysiological networks. This in- 
terpretation is depicted graphically in figure 4.6 us- 
ing colors corresponding to those in the imaging 
results. Results are divided into six domains con- 
taining one or more networks: (1) vigilance, (2) er- 
ror monitoring and inhibition, (3) motor, (4) visual, 
(5) higher order visual/motor, and (6) visual moni- 
toring. Components are grouped according to their 
modulation by driving with the speed-modulated 
components indicated as well. In this figure, we 
also show additional refinements that have been 
made in a subsequent analysis involving alcohol in- 
toxication (Calhoun, Pekar, & Pearlson, 2004). 
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Figure 4.5. Components demonstrating significant speed-related modulation. The frontoparietal component (a) 
demonstrated decreases during the driving epoch and greater decreases for the faster drivers. The anterior cingu- 
late/orbitofrontal component (b) decreased more rapidly for the faster drivers. The lines connect the three subjects 
who were scanned during both faster and slower driving. 
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Error Monitoring 
& Inhibition 

motivation 
risk assessment 
"internal space" 



Exponential decrease 
during driving 






Low-Order Visual 










High-Order Visual 

visuomotor integration 


Increased during driving 
Increased less during watching 



Vigilance 

spatial attention, monitoring 
dorsal visual stream 
ventral visual stream 
"external space" 



Decreased only during driving 




Motor Control 

gross motor control 
motor planning 



Motor Coordination 

fine motor control 
visuomotor integration 



Increased only during driving 



Visual Monitoring 



Increased during epoch 
transitions 



Figure 4.6. Interpretation of imaging results involved in driving. The colors correspond to those used 
in figure 2.4. Components are grouped according to the averaged pattern demonstrated by their time 
courses. The speed-modulated components are indicated with arrows. See also color insert. 



Discussion 

A previous fMRI study involving simulated aviation 
found speed-related changes in frontal areas similar 
to those that we have observed (Peres et al., 2000). 
Attentional modulation may explain the speed- 
related changes as these two components (blue, 
pink) include areas implicated in both the anterior 
and posterior divisions of the attention system 
(Posner, DiGirolamo, & Fernandez-Duque, 1997; 
Schneider, Pimm-Smith, & Worden, 1994). Or- 
bitofrontal cortex has been demonstrated to exhibit 
fMRI signal change during breaches of expectation 
(i.e., error detection) in a visual task (Nobre, Coull, 
Frith, & Mesulam, 1999). Our finding of anterior 
cingulate cortex in the pink component and both 
anterior and posterior cingulate cortex in the blue 
component is consistent with recent studies demon- 
strating functionally distinct anterior and posterior 
cingulate regions in spatial attention (Mesulam, No- 
bre, Kim, Parrish, & Gitelman, 2001). The frontal 
and parietal regions identified in the blue compo- 
nent have also been implicated in attentional tasks 
involving selected and divided attention (Corbetta, 



Miezin, Dobmeyer, Shulman, & Petersen, 1991; 
Corbetta et al., 1998). The angular gyrus, superior 
parietal gyrus, and posterior cingulate gyrus were 
also identified in the simulated aviation task. In our 
study, these areas were contained within the same 
component and are thus functionally connected to 
one another; that is, they demonstrate similar fMRI 
signal changes, and in that sense are distinct from 
areas involved in other components. 

It is also informative to consider the compo- 
nents identified in the context of their interactions. 
The overlapping areas of the blue and pink compo- 
nents, consisting mainly of portions of the anterior 
cingulate and the medial frontal gyrus, are indi- 
cated in figure 4.4 and table 4.1. The anterior cin- 
gulate has been divided into rostral affect and 
caudal cognition regions (Devinsky, Morrell, & 
Vogt, 1995), consistent with the division between 
the pink and blue components. Note that activity 
in the blue regions decreases rapidly during the 
driving epoch, whereas the pink regions decrease 
slowly during the driving condition. One interpre- 
tation of these results is that a narrowing of atten- 
tion (vigilance) is initiated once the driving 
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condition begins. Error correction and disinhibi- 
tion are revealed as a gradual decline of this com- 
ponent at a rate determined in part by the vigilance 
network. During the fast driving condition, the 
vigilance component changes more; thus the error 
correction and disinhibition component decreases 
at a faster rate. An EEG study utilizing the Need for 
Speed simulation software revealed greater alpha 
power in the frontal lobes during replay than dur- 
ing driving and was interpreted as being consistent 
with a reduction of attention during the replay task 
(Schier, 2000). Our results are consistent with this 
interpretation, as neither error monitoring nor fo- 
cused vigilance is presumably prominent during 
replay (watching). 

The inverse correlation between speed and 
frontal/cingulate activity suggests an alternative in- 
terpretation. Driving fast should engage reflex or 
overlearned responses more than critical reasoning 
resources. As the cingulate gyrus has been claimed 
to engage in the monitoring of such resources, 
such a view implies that the cingulate gyrus should 
be less activated as well. Such interpretations, 
while consistent with our results, require further 
investigation. For example, it would be interesting 
to study the effect of speed at more than two levels, 
in a parametric manner. In addition, physiological 
measures such as galvanic skin conductance could 
be used to test for correlations with stress levels. 

Other activated components we observed 
were consistent with prior reports. For example, 
the visual association/cerebellar component (white) 
demonstrates activation in regions previously found 
to be involved in orientation (Allen, Buxton, Wong, 
& Courchesne, 1997) and complex scene inter- 
pretation or memory processing (Menon, White, 
Eliez, Glover, & Reiss, 2000). This component also 
appears to contain areas involved in the modula- 
tion of connectivity between primary visual (V2) 
and motion-sensitive visual regions (V5/area MT), 
such as parietal cortex (Brodmann area 7), along 
with visual association areas (Friston & Buchel, 
2000). These areas, along with the primary visual 
areas (yellow), have been implicated in sensory 
acquisition (Bower, 1997) and attention or antici- 
pation (Akshoomoff, Courchesne, & Townsend, 
1997). Activation in both the white and yellow 
components was increased above fixation during 
watching and further increased during driving. 
This is in contrast to Walter et al. (2001), but is 
consistent with sensory acquisition (present in 



both driving and watching) combined with the 
attentional and motor elements of driving (present 
in the driving epoch). That is, the further increase 
in these areas during driving appears to be an at- 
tentional modulated increase (Gandhi, Heeger, & 
Boynton, 1999). 

The transient visual areas (green) demonstrate 
an increase at the transitions between epochs. We 
identified similar areas, also transiently changing 
between epochs, in a simple visual task (Calhoun, 
Adali, Pearlson, & Pekar, 2001b). Similar areas 
have been detected in a meta-analysis of transient 
activation during block transitions (Konishi, Don- 
aldson, & Buckner, 2001) and may be involved in 
switching tasks in general. The red component was 
mostly in the cerebellum in areas implicated in 
motor preparation (Thach, Goodkin, & Keating, 
1992). Primary motor contributions were low in 
amplitude, presumably due to the small amount of 
motor movement involved in controlling the driv- 
ing task (0.1/0.9 Hz rate on average for left or right 
hand, no significant difference with speed). This 
would also explain why there was little activation 
during the watching epoch as during this time mo- 
tor preparation and visuomotor integration are 
presumably minimal. 

The goal of the example fMRI study described in 
this chapter was to capture the temporal nature of 
the neural correlates of the complex behavior of 
driving. We decomposed the activation due to a 
complex behavior into interpretable pieces using a 
novel, generally applicable approach, based upon 
ICA (see figure 4.6). Several components were iden- 
tified, each modulated differently by our imaging 
paradigm. Regions that increased or decreased con- 
sistently, increased transiently, or which exhibited 
gradual signal decay during driving were identified. 
Additionally, two of the components in regions im- 
plicated in vigilance and error monitoring or inhibi- 
tion processes were significantly associated with 
driving speed. Imaging results are grouped into cog- 
nitive domains based upon the areas recruited and 
their modulation with our paradigm. 

In summary, the combination of driving simu- 
lation software and an advanced analytic technique 
enabled us to study simulated driving and develop 
a model for simulated driving and brain activation 
that did not previously exist. It is clear that driving 
is a complex task. The ability to study, with fMRI, a 
complex behavior such as driving, in conjunction 
with paradigms studying more specific aspects of 
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cognition, may enhance our overall understanding 
of the neural correlates of complex behaviors. 
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Main Points 

1 . Functional MRI provides an unprecedented 
view of the living human brain. 

2. The use of fMRI paradigms involving 
naturalistic behaviors has proven challenging 
due to difficulties in software design and 
analytic approaches. 

3. Current simulation environments can be 
imported into the fMRI environment and used 
as a paradigm. 

4. Simulated driving is one such paradigm, with 
important implications for both brain research 
and the understanding of how the impaired 
brain functions in a real- world environment. 

5. Advanced analytic approaches, including inde- 
pendent component analysis, are enabling the 
analysis and interpretation of the complex data 
sets generated during naturalistic behaviors. 



Key Readings 

Calhoun, V D., Adali, T., Pearlson, G. D., & Pekar, J.J. 
(2001). A method for making group inferences 
from functional MRI data using independent com- 
ponent analysis. Human Brain Mapping, 14, 
140-151. 

Calhoun, V D., Pekar, J. J., & Pearlson, G. D. (2004). 
Alcohol intoxication effects on simulated driving: 
Exploring alcohol-dose effects on brain activation 
using functional MRI. Newopsychopharmacology , 
29, 2097-2107. 

McKeown, M. J., Makeig, S., Brown, G. G.Jung, S. S., 
Kindermann, S., Bell, A. J., et al. (1998). Analysis 
of fMRI data by blind separation into independent 
spatial components. Human Brain Mapping, 6, 
160-188. 

Turner, R., Howseman, A., Rees, G. E., Josephs, O., & 
Friston, K. (1998). Functional magnetic reso- 
nance imaging of the human brain: Data acquisi- 
tion and analysis. Experimental Brain Research, 
123, 5-12. 



References 

Akshoomoff, N. A., Courchesne, E., & Townsend, J. 
(1997). Attention coordination and anticipatory 
control. International Review oj Neurobiology, 41, 
575-598. 

Allen, G, Buxton, R. B., Wong, E. G, & Courchesne, 
E. (1997). Attentional activation of the cerebellum 
independent of motor involvement. Science, 275, 
1940-1943. 

Ballard, D. H., Hayhoe, M. M., Salgian, G, & Shinoda, 
H. (2000). Spatio-temporal organization of behav- 
ior. Spatial Vision, 13, 321-333. 

Bell, A. J., & Sejnowski, T. J. (1995). An information 
maximization approach to blind separation and 
blind deconvolution. Neural Computing, 7, 
1129-1159. 

Biswal, B. B., & Ulmer, J. L. (1999). Blind source 
separation of multiple signal sources of fMRI data 
sets using independent components analysis. 
Journal oj Computer Assisted Tomography, 23, 
265-271. 

Blumer, D., & Benson, D. F (1975). Psychiatric aspects 
of neurologic diseases. In D. E Benson & D. 
Blumer (Eds.), Personality changes with frontal and 
temporal lobe lesions (pp. 151-170). New York: 
Grune and Stratton. 

Bower, J. M. (1997). Control of sensory data acquisition. 
International Review oj Neurobiology, 41, 489-513. 

Cabeza, R., & Nyberg, L. (2000). Imaging cognition II: 
An empirical review of 275 PET and fMRI studies. 
journal oj Cognitive Neuroscience, 12, 1-47. 

Calhoun, V D., Adali, T, Hansen, J. C, Larsen, J., & 
Pekar, J.J. (2003). ICA of fMRI: An overview. In 
Proceedings oj the International Conference on ICA 
and BSS, Nara, Japan. 

Calhoun, V D., Adali, T, Kraut, M., & Pearlson, G D. 
(2000). A weighted-least squares algorithm for es- 
timation and visualization of relative latencies in 
event-related functional MRI. Magnetic Resonance 
in Medicine, 44, 947-954. 

Calhoun, V D., Pekar, J. J., & Pearlson, G D. (2004). 
Alcohol intoxication effects on simulated driving: 
Exploring alcohol-dose effects on brain activation 
using functional MRI. Newopsychopharmacology, 
29, 2097-2107. 

Calhoun, V D., Adali, T, Pearlson, G.D,& Pekar, J. J. 
(2001a). A method for making group inferences 
from functional MRI data using independent com- 
ponent analysis. Human Brain Mapping, 14, 
140-151. 

Calhoun, V D., Adali, T, Pearlson, G.D,& Pekar, J.J. 
(2001c). Spatial and temporal independent com- 
ponent analysis of functional MRI data containing 
a pair of task-related waveforms. Human Brain 
Mapping, 13, 43-53. 



fMRI: Advanced Methods and Applications to Driving 



63 



Calhoun, V D., Adali, T., Stevens, M., Kiehl, K. A., & 
Pekar, J. J. (2005). Semi-blind ICA of fMRI: A 
method for utilizing hypothesis-derived time 
courses in a spatial ICA analysis. Neuroimage, 25, 
527-538. 

Corbetta, M., Akbudak, E., Conturo, T. E., Snyder, 
A. Z., Ollinger, J. M., Drury H. A., et al. (1998). 
A common network of functional areas for atten- 
tion and eye movements. Neuron, 21, 761-773. 

Corbetta, M., Miezin, E M., Dobmeyer, S., Shulman, 
G. L., & Petersen, S. E. (1991). Selective and di- 
vided attention during visual discriminations of 
shape, color and speed: Functional anatomy by 
positron emission tomography. Journal ojNeuro- 
science, 11,2383-2402. 

Devinsky, O., Morrell, M.J., &Vogt, B. A. (1995). 
Contributions of anterior cingulated cortex to be- 
haviour. Brain, iiS(Pt. 1), 279-306. 

DuannJ. R.Jung, T. P., Kuo, W J., Yeh, T. C, Makeig, 
S., Hsieh, J. C, et al. (2002). Single-trial variability 
in event-related BOLD signals. Neuroimage, 15, 
823-835. 

Electronic Arts. (1998). Need for Speed 11 [computer 
driving game] . Redwood City, CA: Author. 

Frackowiak, R. S. J., Friston, K. J., Frith, C, Dolan, R., 
Price, P, Zeki, S., et al. (2003). Human brain junc- 
tion. New York: Academic Press. 

Friston, K. J., & Buchel, C. (2000). Attentional modu- 
lation of effective connectivity from V2 to V5/MT 
in humans. Proceedings of the National Academy of 
Sciences USA, 97, 7591-7596. 

Friston, K. J., Fletcher, P., Josephs, O., Holmes, A., 
Rugg, M. D., & Turner, R. (1998). Event-related 
FMRI: Characterizing differential responses. Neu- 
roimage, 7, 30-40. 

Gandhi, S. P, Heeger, D. J., & Boynton, G. M. (1999). 
Spatial attention affects brain activity in human 
primary visual cortex. Proceedings oj the National 
Academy oj Sciences USA, 96, 3314-3319. 

Glover, G. H. (1999). Deconvolution of impulse re- 
sponse in event-related BOLD fMRI. Neuroimage, 
9, 416-429. 

Groeger, J. (2000). Understanding driving: Applying cog- 
nitive psychology to a complex everyday task. New 
York: Psychology Press. 

Holmes, A. P, & Friston, K.J.. (1998). Generalizability 
random effects, and population inference. Neuro- 
image, 7, S754. 

Hopfmger, J. B., Buonocore, M. H, & Mangun, G. R. 
(2000). The neural mechanisms of top-down at- 
tentional control. Nature Neuroscience, 3, 
284-291. 

Janca, A., Ustun, T. B., & Sartorius, N. (1994). New 
versions of World Health Organization instru- 
ments for the assessment of mental disorders. Acta 
Psychiatrica Scandinavica, 90, 73-83. 



Komshi, S., Donaldson, D. L, & Buckner, R. L. (2001). 
Transient activation during block transition, Neu- 
roimage, 13, 364-374. 

McGinty V B., Shih, R. A., Garrett, E. S., Calhoun, 
V D., & Pearlson, G. D. (2001). Assessment of in- 
toxicated driving with a simulator: A validation 
study with on road driving. In Proceedings oj the 
Human Centered Transportation and Simulation Con- 
ference, November 4-7, 2001; Iowa City, IA. 

McKeown, M. J., Hansen, L. K., & Sejnowski, T. J. 
(2003). Independent component analysis of 
functional MRI: What is signal and what is noise? 
Current Opinion in Neurobiology, 13, 620-629. 

McKeown, M. J., Makeig, S., Brown, G. G.Jung, T. P, 
Kindermann, S. S., Bell, A. J., et al. (1998). Analy- 
sis of fMRI data by blind separation into indepen- 
dent spatial components. Human Brain Mapping, 6, 
160-188. 

Menon, V, White, C. D., Eliez, S., Glover, G. H, & 
Reiss, A. L. (2000). Analysis of a distributed neural 
system involved in spatial information, novelty, 
and memory processing. Human Brain Mapping, 
11, 117-129. 

Mesulam, M. M., Nobre, A. C, Kim, Y. H, Parrish, 
T. B., & Gitelman, D. R. (2001). Heterogeneity of 
cingulated contributions to spatial attention. Neu- 
roimage, 13, 1065-1072. 

Nobre, A. C, CoullJ. T, Frith, C. D., & Mesulam, 
M. M. (1999). Orbitofrontal cortex is activated 
during breaches of expectation in tasks of visual 
attention. Nature Neuroscience, 2, 11-12. 

Peres, M., Van de Moortele, P E, Pierard, C, Lehericy, 
S., LeBihan, D., & Guezennez, C. Y (2000). Func- 
tional magnetic resonance imaging of mental strat- 
egy in a simulated aviation performance task. 
Aviation, Space and Environmental Medicine, 71, 
1218-1231. 

Posner, M. L, DiGirolamo, G J., & Fernandez-Duque, 
D. (1997). Brain mechanisms of cognitive skills. 
Consciousness and Cognition, 6, 267-290. 

Ranney, T. A. (1994). Models of driving behavior: A re- 
view of their evolution. Accident Analysis and Pre- 
vention, 26, 733-350. 

Rauch, S. L. Jenike, M. A., Alpert, N. M., Baer, L., Bri- 
eter, H. C, Savage, C. R., et al. (1994). Regional 
cerebral blood flow measured during symptom 
provocation in obsessive-compulsive disorder us- 
ing oxygen 15-labeled carbon dioxide and 
positron emission tomography. Archives oj General 
Psychiatry, 51, 62-70. 

Rees, G. (2001). Neuroimaging of visual awareness in 
patients and normal subjects. Current Opinion in 
Neurobiology, 11, 150-156. 

Schier, M. A. (2000). Changes in EEG alpha power 
during simulated driving: A demonstration. Inter- 
national journal oj Psychophysiology, 37, 155-162. 



64 



Neuroergonomics Methods 



Schneider, W, Pimm-Smith, M., & Worden, M. 

(1994). Neurobiology of attention and automatic- 
ity. Current Opinion in Neurobiology, 4, 177-182. 

Talairach, J., & Tournoux, P (1988). A co -planar stereo- 
taxic atlas of a human brain. Stuttgart: Thieme. 

Thach, W T, Goodkm, H. P, & Keating, J. G. (1992). 
The cerebellum and the adaptive coordination of 
movement. Annual Review ojNeuroscience, 15, 
403-442. 

Van de Moortele, P E, Cerf, B., Lobel, E., Paradis, A. L.. 
Faurion, A., & Le Bihan, D. (1997). Latencies in 



FMRI time-series: Effect of slice acquisition order 
and perception. NMR Biomedicine, 10, 230-236. 

Walter, H., Vetter, S. C, Grothe, J., Wunderlich, A. P, 
Hahn, S., & Spitzer, M. (2001). The neural corre- 
lates of driving. Neuroreport, 12, 1763-1767. 

Woods, R. P. (1996). Modeling for intergroup 
comparisons of imaging data. Neuroimage, 4, 
S84-S94. 

Worsley, K. J., & Friston, K. J. (1995). Analysis of 
fMRI time-series revisited — again. Neuroimage, 7, 
30-40. 




Gabriele Gratton and Monica Fabiani 



Optical Imaging of Brain Function 



In this chapter, we review the use of noninva- 
sive optical imaging methods in studying human 
brain function, with a view toward their possible 
applications to neuroergonomics. Fast optical im- 
aging methods make it possible to image brain ac- 
tivity with subcentimeter spatial resolution and 
millisecond-level temporal resolution. In addition, 
these methods are also relatively inexpensive and 
adaptable to different experimental situations. These 
properties make optical imaging a promising new 
approach for the study of brain function in experi- 
mental and possibly applied situations. The main 
limitations of optical imaging methods are their 
penetration (a few centimeters from the surface of 
the head) and, at least at present, a relatively low 
signal-to-noise ratio. 

Optical imaging methods are a class of tech- 
niques that investigate the way light interacts 
with tissue. If the tissue is very superficial, light of 
many wavelengths can be used for the measure- 
ments. However, if the tissue is deep, like the 
brain, only a narrow wavelength range is usable 
for measurement — a window between 680 and 
1,000 nm (commonly called the near-infrared or 
NIR range). This is because at other wavelengths 
water and hemoglobin absorb much of the light, 
whereas in the NIR range these substances absorb 



relatively little. Near- infrared photons can pene- 
trate deeply into tissue (up to 5 cm). The main ob- 
stacle to the movement of NIR photons through 
the head is represented by the fact that most head 
tissues (such as the skull and gray and white mat- 
ter) are all highly scattering. Thus, the movement 
of photons through the head can be represented as 
a diffusion process, and noninvasive optical imag- 
ing is sometimes called diffusive optical imaging. 
Besides limiting the penetration of the photons, 
the scattering process also limits the spatial resolu- 
tion of the technique to a few millimeters. 

Work conducted in our laboratory and others 
during the last few years has shown that optical 
imaging can be used to study the time course of ac- 
tivity in specific cortical areas. We were the first to 
show that noninvasive optical imaging is sensitive 
to neuronal activity (Gratton, Corballis, Cho, Fabi- 
ani, & Hood, 1995). This result has been repli- 
cated many times in our laboratory (for a review, 
see Gratton & Fabiani, 2001), and in at least four 
other laboratories (Franceschini, & Boas, 2004; 
Steinbrink et al., 2000; Tse, Tien, & Penney, 2004; 
Wolf, Wolf, Choi, Paunescu, 2003; Wolf, Wolf, 
Choi, Toronov, et al., 2003). In a number of differ- 
ent studies, we have found that this phenomenon 
can be observed in all cortical areas investigated 
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(visual cortex, auditory cortex, somatosensory cor- 
tex, motor areas, prefrontal and parietal cortex). 
Further, we have developed a technology for full- 
head imaging (using 1,024 channels) that has been 
successfully applied to cognitive paradigms. We 
have also developed a full array of recording and 
analysis tools that make noninvasive cortical imag- 
ing a practical method for research and potentially 
for some applied settings. In the remainder of this 
chapter, we review the main principles of optical 
imaging and some of the types of optical signals 
that can be used to study brain function, as well as 
aspects of recording and analysis and examples of 
applications to cognitive neuroscience issues. 



Principles of Noninvasive 
Optical Imaging 

Noninvasive optical imaging is based on two major 
observations: 

1. Near-infrared light penetrates several centime- 
ters into tissue. Because of the scattering prop- 
erties of most head tissues and the size of the 
adult human head, in the case of noninvasive 
brain imaging this process approximates a dif- 
fusion process. 

2. The parameters of this diffusion process are in- 
fluenced by physiological events in the brain. 
These parameters are related to changes in the 
scattering and absorption properties of the tis- 
sue itself. 

Noninvasive optical imaging studies are based 
on applying small, localized sources of light to 
points on the surface of the head and measuring the 
amount or delay of the light reaching detectors lo- 
cated also on the surface at some distance (typically 
a few centimeters). Note that, because of the diffu- 
sion process, this procedure is sensitive not only to 
superficial events but also to events occurring at 
some depth in the tissue. To understand this phe- 
nomenon, consider that the photons emitted by the 
sources will propagate in a random fashion through 
the tissue. Because the movement of the photons is 
random, there is no preferred direction, and if the 
medium was infinite and homogenous, the move- 
ment of the photons could describe a sphere. How- 
ever, in reality the diffusive tissue (e.g., the head) is 
surrounded by a nondiffusive medium (e.g., air). In 
this situation, photons that are very close to the sur- 



face are likely to move outside the diffusive 
medium. Once they enter the nondiffusive medium 
(air), their movement becomes rectilinear. There- 
fore, they are not likely to enter the head again and 
will not reach the detector. This means that only 
photons traveling at some depth inside the tissue 
are likely to reach the detector. For this reason, dif- 
fusive optical imaging methods (also called photon 
migration methods) become sensitive to changes in 
optical properties that occur at some depth inside 
the tissue (see Gratton, Maier, Fabiani, Mantulin, & 
Gratton, 1994). The depth investigated by a particu- 
lar source-detector configuration depends on many 
parameters, one of the most important of which is 
the source-detector distance (see figure 5.1; see also 
Gratton, Sarno, Maclin, Corballis, & Fabiani, 2000). 
Currently, two main families of methods for 
noninvasive optical imaging are in use: (a) simpler 
methods based on continuous light sources (con- 
tinuous wave or CW methods); and (b) more elab- 
orate techniques based on light sources varying in 
intensity over time (time-resolved or TR methods). 
Although more complex and expensive, TR meth- 
ods provide important additional information 
about the movement of photons through tissue, 
and specifically the ability to measure the photons' 
time-of-flight (i.e., the time required by photons 
to move between a source and a detector). This 
parameter is very important for the accurate mea- 
surement of optical properties (scattering and ab- 
sorption) of the tissue and, in our observations, has 
been particularly useful for studying fast changes 
in optical parameters presumably related to neu- 
ronal activity. 



Optical Signals 

A large body of data supports the idea that at least 
two categories of signals can be measured with 
noninvasive optical methods: (a) fast signals, oc- 
curring simultaneously with neuronal activity; and 
(b) slow signals, mostly related to the changes in 
the hemodynamic and metabolic properties of the 
tissue that lag neuronal activity of several seconds. 
Fast signals are related to changes in the scatter- 
ing of neural tissue that occur simultaneously with 
electrical events related to neuronal activity. The 
first demonstration of these scattering changes was 
obtained by Hill and Keynes (1949). Subsequent 
studies have confirmed these original findings, 
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Figure 5.1 . Mathematical model of the propagation of light into a scattering medium. Top left, diffusion of light in the scattering medium from a surface source; top middle, 
area of diffusion for photons produced by a surface source and reaching a source detector; bottom left, effect of a boundary between the scattering medium (such as the head) 
and a nonscattering medium (such as air); bottom middle, area of diffusion for photons produced by a surface source and reaching a source detector, including effects due to 
boundary between scattering and nonscattering medium, simulating the actual situation in human head; right, schematic representation of variations in the depth of the diffu- 
sion spindle as a function of variations in source-detector distance. See also color insert. 
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first in isolated neurons (Cohen, Hille, Keynes, 
Landowne, & Rojas, 1971), then in vitro in hip- 
pocampal slices (Frostig, Lieke, Ts'o, & Grinvald, 
1990), and finally in the central nervous system 
of living invertebrates (Stepnoski et al., 1991) 
and mammals (Rector, Poe, Kristensen, & Harper, 
1997). The evidence that this phenomenon can 
also be observed noninvasively in humans is re- 
viewed in a later section. These scattering changes 
are assumed to be directly associated with post- 
synaptic depolarization and hyperpolarization 
(MacVicar & Hochman, 1991). Two phenomena 
probably contribute to these scattering changes: (a) 
cell swelling and shrinking, due to the movement 
of water in and out of the cell (associated with the 
movement of ions; MacVicar & Hochman, 1991); 
and (b) membrane deformation, associated with 
reorientation of membrane proteins and phospho- 
lipids (Stepnoski et al., 1991). Which of these two 
phenomena is most critical is still unclear. The evi- 
dence suggests that, at least at the macro level rele- 
vant for noninvasive imaging, the fast signal can be 
considered as a scalar, isotropic change in the scat- 
tering properties of a piece of nervous tissue, with 
decrease in scattering related to excitation of a par- 
ticular area, and increase in scattering related to in- 
hibition of the same area (Rector et al., 1997). 

The results obtained by Rector et al. (1997) 
with fibers implanted in the rat hippocampus led 
to specific predictions about the type of phenome- 
non that should be observed using sources and de- 
tectors on the same side of the relevant tissue, as is 
the case with noninvasive cortical measures in the 
adult human brain. In fact, cortical excitation should 
be associated with an increase in the average pho- 
ton time of flight (because, on average, the photons 
penetrate deeper into the tissue), and cortical inhi- 
bition should be associated with a reduction in the 
average time of flight (because, on average, the 
photons penetrate less into the tissue). As shown 
later, this is exactly the result we typically obtain in 
our studies. 

It should be noted that these predictions only 
hold for situations in which the activated area is at 
least 5-10 mm deep, because at shallower depths 
light propagation may not follow a diffusive regime. 
It should further be noted that these predictions are 
related to TR measurements. In fact, fast optical ef- 
fects can also be recorded with CW methods (as 
demonstrated by Steinbrink et al, 2000; and by 
Franceschini & Boas, 2004), typically as increases 



in the transparency of active cortical areas. How- 
ever, intensity estimates (the only ones available 
with CW methods) are more sensitive to superficial 
(nonbrain) effects and to slow hemodynamic effects 
(see below) than time-of-flight measures (available 
only with TR methods). Therefore, in our experi- 
ence, TR methods appear to be particularly suited 
for the study of fast optical signals. In any case, fast 
effects are relatively small and typically require ex- 
tensive averaging for their measurement. 

Intrinsic changes in the coloration of the brain 
due to variations in blood flow in active brain areas 
have been shown in an extensive series of studies on 
optical imaging of exposed animal cortex conducted 
in the last 20 years (see Frostig et al., 1990; Grin- 
vald, Lieke, Frostig, Gilbert, & Wiesel, 1986). The 
intrinsic optical signal is typically observed over 
several seconds after the stimulation of a particular 
cortical area. This signal capitalizes in part on dif- 
ferences in the absorption spectra of oxy- and de- 
oxyhemoglobin. Using a spectroscopic approach, 
Malonek and Grinvald (1996) demonstrated three 
major types of optical changes that develop over 
several seconds after stimulation of a particular cor- 
tical area: (a) a relatively rapid blood deoxygenation 
occurring during the first couple of seconds (this 
signal has been associated with the initial dip in the 
functional magnetic resonance imaging (fMRI) sig- 
nal reported by Menon et al., 1995); (b) a large in- 
crease in oxygenation occurring several seconds 
after stimulation (the drop in deoxyhemoglobin 
concentration associated with this signal corre- 
sponds to the blood oxygenated level-dependent 
[BOLD] effect observed with fMRI); and (c) changes 
in scattering beginning immediately after the cortex 
is excited and lasting until the end of stimulation, 
whose nature is not completely understood (these 
changes in scattering may include the fast effects re- 
ported above). Of these three signals, the oxygena- 
tion effect has been first reported in noninvasive 
imaging from the human cortex by Hoshi and 
Tamura (1993) and subsequently replicated in a 
large number of studies (for an early review of this 
work, see Villringer & Chance, 1997). The other 
slow optical signals are more difficult to observe 
noninvasively, probably because they are dwarfed by 
the large oxygenation effect. Slow effects are easy to 
observe sometimes even on single trials. They can be 
detected using both CW and TR methods, although 
TR methods provide a more precise quantification of 
oxy- and deoxyhemoglobin concentration. 



Optical Imaging of Brain Function 



69 



In summary, two types of signals can be de- 
tected using noninvasive optical imaging methods: 
(a) fast scattering effects related to neuronal activ- 
ity, and (b) slow absorption effects related to hemo- 
dynamic changes. These signals are reviewed in 
more details in the following sections. 

Fast Optical Signals 

The Event-Related Optical Signal 

During the last decades, our laboratory has carried 
out a large number of studies demonstrating the 
possibility of detecting the fast optical signal with 
noninvasive methods. Our initial observations were 
in response to visual stimulation (Gratton, Corballis, 
et al., 1995; for a replication, see Gratton & Fabiani, 
2003). This work, using a frequency-domain TR 
method, revealed the occurrence of a fast optical sig- 
nal marked by an increase in the photons' time-of- 
flight parameter (also called phase delay, because it 
is obtained by measuring the phase delay of the 
photon density wave of NIR light modulated at 
high — MHz range — frequencies). This study was 
based on the stimulation (obtained through pattern 
reversals) of each of the four quadrants of the visual 
field in different blocks. Recordings were obtained 
from an array of locations over the occipital cortex. 
The prediction was made that each stimulation con- 
dition should generate a response in a different set of 
locations, following the contralateral inverted repre- 



sentation of the visual field in the occipital cortex. In 
this way, for each location of the occipital cortex, we 
could compare the optical response obtained when 
that region was expected to be stimulated with the 
optical response obtained when other regions were 
expected to be stimulated. This paradigm controlled 
for a number of possible alternative explanations for 
the results (i.e., nonspecific brain changes, move- 
ments, or luminance artifacts). In addition, it pro- 
vided an initial estimate of the spatial resolution of 
the technique, since it implied that the optical re- 
sponse was limited to the stimulated cortical area. 
The original article showed a clear differentiation be- 
tween the four responses (Gratton, Corballis, et al., 
1995), but was based on a small number of subjects, 
due to the fact that the recording equipment avail- 
able at the time was based on one channel, and 12 
sessions were required to derive a map for one sub- 
ject. For these reasons, we later replicated this study 
using multichannel equipment and a larger num- 
ber of subjects (Gratton & Fabiani, 2003). In this 
second study, we also increased the temporal sam- 
pling (from 20 to 50 Hz). Thus, we could determine 
that the peak latency of the response in visual (pos- 
sibly VI) cortex (which occurred between 50 and 
100 ms in the original study) is in fact close to 
80 ms — corresponding to the peak latency of the CI 
component of the visual evoked potential (see figure 
5.2). Further, when fMRI and EROS data from the 
same subjects are compared, it is apparent that there 
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Figure 5.2. Left panel: Time course of the fast optical response (EROS) observed at the predicted (thick line) and 
control (thin line) locations in Gratton and Fabiani (2003). The error bars indicate the standard error of estimate 
computed across subjects. Right panel: Time course of the fast optical response (EROS) for locations with maxi- 
mum response (thick line) and, from the same locations, when the opposite visual field quadrant was stimulated 
(thin line). From Gratton, G, Corballis, R M., Cho, E., Fabiani, M., & Hood, D.C. (1995). Shades of gray matter: 
Noninvasive optical images of human brain responses during visual stimulation. Psychophysiology, 32, 505-509. 
Reprinted with permission from Blackwell Publishing. 
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Figure 5.3. Comparison of fMRI and EROS responses in a visual stimulation experiment. The data refer to stimu- 
lation of the upper left quadrant of the visual field in one subject. The upper panel reports the BOLD-fMRI slice 
with maximum activity in this condition. Voxels in yellow are those exhibiting the most significant BOLD response. 
The green rectangle corresponds to the area explored with EROS. Maps of the EROS response before stimulation 
and 100 ms and 200 ms after grid reversal are shown at the bottom of the figure, with areas in yellow indicating lo- 
cations of maximum activity. Note that the location of maximum EROS response varies with latency: The earlier re- 
sponse (100 ms) corresponds in location to the more medial fMRI response, whereas the later response (200 ms) 
corresponds in location to the more lateral fMRI response. The data are presented following the radiological con- 
vention (i.e., the brain is represented as if it were seen from the front). See also color insert. 



is a good spatial correspondence between the two 
maps, with EROS further allowing for temporal dis- 
crimination for the order of events (see figure 5.3). 

In the last few years, we have replicated this ba- 
sic finding (i.e., localized increase in phase delay in- 
dicating the activation of a particular cortical area) 
several times in the visual modality (Fabiani, Ho, 
Stinard, & Gratton, 2003; Gratton, 1997; Gratton, 



Fabiani, Goodman-Wood, & DeSoto, 1998; Grat- 
ton, Goodman-Wood, & Fabiani, 2001) using stim- 
uli varying along a number of dimensions, including 
frequency, size, shape, onset, and so on. Further, we 
showed that similar effects are observed in prepara- 
tion for motor responses (De Soto, Fabiani, Geary, & 
Gratton, 2001; Gratton, Fabiani, et al., 1995), as 
well as in response to auditory (Fabiani, Low, Wee, 
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Figure 5.4. Schematic representation of the backscattering of photons under conditions of rest and activity 
in the cortex. Reprinted with permission from Blackwell Publishing. 



Sable, & Gratton, 2006; Rinne et al., 1999; Sable 
et al, 2006) and somatosensory (Maclin, Low, Sable, 
Fabiani, & Gratton, 2004) stimuli. The feasibility of 
recording fast optical signals noninvasively has also 
been reported by four other laboratories: Frances- 
chini and Boas (2004) using motor and somatosen- 
sory modalities, Steinbrink et al. (2000) using 
somatosensory stimulation, Wolf and colleagues us- 
ing both visual (Wolf, Wolf, Choi, Paunescu, 2003; 
Wolf, Wolf, Choi, Toronov, et al., 2003) and motor 
modalities (Morren et al., 2004), and Tse et al. 
(2004) using auditory stimulation. In contrast to 
this positive evidence, only one report of difficulties 
in obtaining a fast optical signal has been reported 
(Syre et al., 2003). 

In summary, the data overwhelmingly support 
the claim that fast optical signals can be recorded 
noninvasively in all modalities that have been at- 
tempted. In general, the signal corresponds to an 
increase in phase delay (available with TR meth- 
ods) and a reduction in light (available with both 
TR and CW methods; see figure 5.4 for a schematic 
illustration). Monte Carlo simulations (presented 



in Gratton, Fabiani, et al, 1995) suggest that the 
increase in phase delay is consistent with a reduc- 
tion in scattering in deep (cortical) areas. This re- 
duction in scattering determines an increase in 
tissue transparency, so that photons on average 
penetrate deeper before reaching the detector. The 
reduction in intensity is to be attributed to the rela- 
tive lack of a reflective surface conducting the pho- 
tons to the detector (see figure 5.4). In this sense, 
the functional signal can be conceptualized as a 
variation of the echo from the cortex of the light 
produced by the source. This conceptualization 
also helps explain why TR methods may be more 
suitable for studying the fast optical signal (al- 
though in some cases CW methods can also detect 
fast signals; see Franceschini & Boas, 2004; Stein- 
brink et al., 2000). 

Because the fast optical signal obtained with 
TR methods is recorded in response to internal or 
external events (and visible in time-locked aver- 
ages, similarly to the event-related brain potential, 
ERP) we have labeled it the event-related optical 
signal (EROS). Our work indicates that EROS can 
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be considered as a measure of neuronal activity in a 
localized cortical area. We have conducted several 
studies to further characterize this signal. These 
studies have showed the following: 

1. EROS activity temporally corresponds with 
ERP activity obtained simultaneously or in 
similar paradigms (e.g., De Soto et al., 2001; 
Gratton et al., 1997, 2001; Rinne et al., 1999). 

2. EROS activity is spatially colocalized with the 
BOLD fMRl response (Gratton et al., 1997, 
2000) and with the slow optical response (see 
below; Gratton et al, 2001). 

3. The amplitude of EROS varies with experimen- 
tal manipulations such as stimulation fre- 
quency in a manner very similar to ERP and 
hemodynamic responses (Gratton et al, 
2001). 

4. The spatial resolution of EROS is approxi- 
mately 0.5-1 cm (Gratton & Fabiani, 2003). 

5. The temporal resolution of EROS is similar to 
that of ERPs, allowing for the identification of 
peaks of activity with a millisecond scale (Grat- 
ton et al., 2001). For example, the peak of the 
auditory Nl is the same as observed in concur- 
rently recorded ERP data (Fabiani et al, 2006). 
The limits of this resolution are determined by 
the characteristics of the neurophysiological 
phenomena under study (i.e., aggregate cell ac- 
tivity rather than action potentials) and by the 
temporal sampling used, rather than by the in- 
herent properties of the signal. 

6. By using multiple source-detector distances, it 
is possible to estimate the depth of EROS re- 
sponses and to show that these estimates cor- 
respond to the depth of activity as measured 
with fMRI (Gratton et al., 2000). These mea- 
sures are possible up to a depth of 2.5 cm 
from the surface of the head. 

These findings, taken together, suggest that EROS 
can be used to study the time course of activity in 
localized cortical areas. Our recent use of a full- 
head recording system promises to make the mea- 
sure a general method for studying brain function. 
Currently, two main limitations should be noted: 

1 . Like all noninvasive optical methods, EROS 
cannot measure deep structures. 

2. With current technology, the signal-to-noise 
ratio of EROS is relatively low, and averaging 



across a large number of trials is required. 
However, application of new methodologies 
(such as the "pi detector," Wolf et al., 2000; a 
high modulation frequency, Toronov et al., 
2004; and appropriate frequency filters, 
Maclin, Gratton, and Fabiani, 2003) may at 
least partially address this problem. 

Applications of EROS in Cognitive Neuroscience 

The combination of spatial and temporal informa- 
tion provided by EROS can be of great utility in 
cognitive neuroimaging. In this area of research, it 
is of great importance to establish not only which 
areas of the brain are active, but the order of their 
activation, as well as the general architecture of the 
information processing system. To illustrate these 
points, we will discuss here the results from two 
cognitive neuroimaging studies we have conducted 
with EROS, both related to concepts of early and 
late levels of selective attention. 

Psychologists have focused on selective at- 
tention for a number of years and have presented 
arguments in favor of or against early and late 
selective attention models (Johnston & Dark, 1986). 
Here we focus on two specific questions: (a) At 
which point within the information processing 
flow is it possible to separate the processing of at- 
tended and unattended items? (b) How long within 
the information processing flow is irrelevant infor- 
mation processed before being discarded? Within a 
cognitive neuroscience framework, the first issue 
can be restated as the identification of the earliest 
response in sensory cortical regions showing mod- 
ulation by attention. The second issue can be cast 
in terms of demonstrating whether simultaneous 
activation of conflicting motor responses is possi- 
ble. We describe two studies showing how EROS 
can be useful to address these questions. 

Locus of Early Selective Attention. ERPs have 
been used for a number of years to address the issue 
of the onset of early selective attention effects (for 
a review, see Hackley 1993). On the basis of ERP 
data, for instance, Martinez et al. (1999) have 
shown that the earliest visually evoked brain activ- 
ity, such as the CI response, is not modulated by at- 
tention, but that attention modulation effects are 
evident in subsequent ERP components, such as the 
PI (see also chapter 3, this volume). These data 
contrast with reports from fMRI studies indicating 
attention effects occurring in Brodmann Area 17 
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(BA 17; primary visual cortex), typically considered 
to be the first cortical station of the visual pathway 
(Gandhi, Heeger, & Boynton, 1999). Martinez et al. 
(1999) proposed that this apparent contrast is de- 
termined by the fact that BA 17 is initially unable to 
separate attended and unattended information, and 
that the attention effects visible in this region are 
due to reafferent activity. They argue that fMRI, 
given its low temporal resolution, would not be 
able to distinguish between the earliest response 
and that associated with the reafferent response. In 
their study, however, their argument was based on 
source modeling of ERP activity, an approach that 
has yet to gain universal acceptance in the field. A 
technique combining spatial and temporal resolu- 
tion would be very useful here, because it would 
allow us to be able to ascertain whether the early 
response in BA 17 is in fact similar for attended 
and unattended stimuli, and that attention modu- 
lation effects become evident only later in pro- 
cessing. With this idea in mind, we (Gratton, 1997) 
recorded EROS in a selective attention task very 
similar to that used by Martinez and colleagues. 
Two streams of stimuli appeared to the left and right 
of fixation, and subjects were instructed to monitor 
one of them for the rare (20%) occurrence of a de- 
viant stimulus (a rectangle instead of a square). 
EROS activity was monitored over occipital areas. 
We found two types of EROS responses, with laten- 
cies between 50 and 150 ms from stimulation — a 
more medial response, presumably from BA 17, 
similar for attended and not attended stimuli, and a 
more lateral response, presumably from BA 18 and 
BA 19, which was significantly larger for attended 
than for unattended stimuli. These data support the 
claim that the initial response in BA 1 7 is unaffected 
by attention, and that early visual selective attention 
effects begin in the extrastriate cortex. 

Parallel Versus Serial Processing. We rephrased 
the question of how far within the processing sys- 
tem irrelevant information is processed in terms of 
evidence for the simultaneous (parallel) activation 
of conflicting motor responses. This question is rel- 
evant to the basic discussion present in cognitive 
psychology about whether the information pro- 
cessing system is organized serially or in parallel. 
Several pieces of evidence (in particular from con- 
flict tasks, such as the Stroop paradigm) have been 
used to argue in favor of the parallel organization of 
the system, but serial accounts for these findings 



have also been presented (Townsend, 1990). Re- 
casting this issue in terms of brain activity may 
make it more easily treatable. Specifically, the ulti- 
mate test of the parallelism of the information pro- 
cessing system would come from the demonstration 
that two output systems, such as the left and right 
motor cortices, can be activated simultaneously, one 
by relevant and the other by irrelevant information. 
This would demonstrate that irrelevant information 
is processed (at least in part) all the way up to the 
response system. Unfortunately, the most com- 
monly used brain imaging methods have had prob- 
lems providing this demonstration. In fact, this 
issue requires concurrent spatial resolution (so that 
activity in the two motor cortices can be measured 
independently) and temporal resolution (so that 
the simultaneous — or parallel — timing of the two 
activations can be ascertained). Therefore, neither 
ERPs nor fMRI can be used to provide this evi- 
dence. We (De Soto et al, 2001) used EROS for this 
purpose. Specifically, we recorded EROS from the 
left and right motor cortex while subjects were per- 
forming a spatial Stroop task. In this task, subjects 
are presented with either the word above or the 
word below, presented above or below a fixation 
cross. In different trials, subjects were asked to re- 
spond (with one or the other hand) on the basis of 
either the meaning or the position of the word. On 
half of the trials, the two dimensions conflicted. If 
parallel processing were possible, on these correct 
conflict trials we expected subjects to activate si- 
multaneously both the motor cortex associated with 
the correct response and that associated with the in- 
correct response (hence the delay in RT). Only uni- 
lateral activation was expected on no-conflict trials. 
This is in fact the pattern of results we obtained. 
These results provide support for parallel models 
of information processing and show that irrelevant 
information is processed all the way up to the re- 
sponse system, concurrently with relevant informa- 
tion. Similarly to the selective attention study, this 
study shows the usefulness of a technique combin- 
ing spatial and temporal information in investigat- 
ing the architecture of cognition. 

Slow Optical Signals 

Slow optical signals are due to the hemodynamic 
changes occurring in active neural areas (see fig- 
ure 5.5 for an example). These changes are well 
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Figure 5.5. Changes in the concentration of oxyhemoglobin (solid black line) and deoxyhemo- 
globin (gray line) in occipital areas during visual stimulation (beginning of stimulation marked by 
the vertical line at time 0) measured with near-infrared spectroscopy. 



demonstrated and are the basis for the BOLD fMRI 
signal and for 15 positron emission tomography 
(PET) as well as for invasive optical imaging in ani- 
mals. In humans, slow optical signals are recorded 
in a number of laboratories and are often referred 
to as near-infrared spectroscopy (NIRS). This is be- 
cause slow signal recordings typically employ mul- 
tiple light wavelengths in the NIR range and 
decompose the signal into changes in oxy- and de- 
oxyhemoglobin using a spectroscopic approach. 
Many of the considerations that can be made about 
BOLD fMRI studies are also valid about the NIRS 
signals. Slow optical signals occur with a latency of 
approximately 4-5 seconds compared to the elicit- 
ing neural activity. The quantitative relationship 
between neuronal and hemodynamic signals (neu- 
rovascular coupling) has been the subject of inves- 
tigation. For instance, we used a visual stimulation 
paradigm in which the frequency of visual stim- 
ulation was varied between 1 and 10 Hz (Gratton 
et al., 2001). We measured both fast and slow opti- 
cal signals concurrently and found that the ampli- 
tude of the slow signal could be predicted on the 
basis of the amplitude of the fast signal integrated 
over time. The relationship between the two sig- 
nals was approximately linear. 

Most studies using NIRS report a relatively low 
spatial resolution for the slow optical signal (2 cm or 
so) and typically attribute this low resolution to the 



diffusive properties of the method. However, the 
recording methodology may play a significant role in 
the low resolution reported. In fact, these studies are 
typically based on a very sparse recording montage, 
so that channels rarely or never overlap with each 
other. Wolf et al. (2000) have shown that a much 
better spatial resolution (and signal-to-noise ratio) 
can be obtained by using a number of partially over- 
lapping channels (pi detectors). Further, very few of 
the NIRS studies published to date take into account 
individual differences in brain anatomy or coregister 
the optical data with anatomical MR images. It is 
therefore likely that the spatial resolution of slow op- 
tical measures can be greatly improved by the use of 
appropriate methodologies. In summary, slow opti- 
cal measures (NIRS) can provide data that are quite 
comparable to other hemodynamic imaging meth- 
ods with the following main differences: 

1. Data from deep structures are not available. 

2. The NIRS technology is more flexible and 
portable and can be applied to many normal 
and even bedside settings. 

When compared to the fast signal (EROS), the 
following differences should be noted: 

1. NIRS is based on slow hemodynamic changes 
and therefore has an intrinsically limited tem- 
poral resolution. 



Optical Imaging of Brain Function 



75 



2. NIRS has a better signal-to-noise ratio than 
EROS, as changes in oxy- and deoxyhemoglo- 
bin can be sometimes seen in individual 
trials. 

It is important to note that fast and slow optical 
signals can be easily recorded simultaneously, and 
therefore there is no need to choose one over the 
other, although experimental conditions can be 
created that optimize the recording conditions for 
one or the other of these methods. 



Methodology 

In this section, we describe some of the methods 
used for recording noninvasive optical imaging 
data. Methods vary greatly across laboratories, so 
for simplicity we focus on the methodology used in 
our lab. 

Recording 

Our optical recording apparatus (Imagent, built by 
ISS Inc., Champaign, IL) uses a frequency-domain 



(FD) TR method and is flexible both in the number 
of sources and detectors that can be used, and in a 
number of other recoding parameters. This equip- 
ment uses laser diodes as light sources, emitting 
light at 690 and 830 nm (note that other types of 
sources, including larger lasers, lamps, or LEDs are 
used by other optical recording devices). Our par- 
ticular Imagent device has 128 possible sources, al- 
though in practice only about half of them are used 
in a particular study because of the need to avoid 
cross talk of different sources activated at the same 
time. The detectors (16 for the Imagent housed in 
our lab) are photomultiplier tubes (PMTs; note that 
other types of detectors are possible and are used 
in other devices, including light-sensitive diodes 
and CCD cameras). Sources and detectors are 
mounted on the head by means of a modified bike 
helmet (see figure 5.6). Care is taken to move the 
hair from under the detector fibers (which are 3- 
mm fiber bundles). Source fibers are small (400 
microns in diameter) and can easily be placed in 
between hair. 

Note that the number of sources greatly exceeds 
the number of detectors. Thus, each detector picks 




Figure 5.6. Photograph of modified helmet used for recording (left panel); examples of digitized montage (top 
right) and surface channel projections (bottom right). From Griffin, R. (1999). Illuminating the brain: The love and 
science of EROS. Illumination, 2(2), 20-23. Used with permission from photographer Rob Hill and the University 
of Missouri-Columbia. See also color insert. 
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up light emitted by different sources. To distin- 
guish which source is providing the light received 
by the detector, sources are time-multiplexed. Al- 
though the use of a large number of sources yields 
a large number of recording channels — which is 
very useful to increase the spatial resolution and 
area investigated by the optical recordings — it has 
the disadvantage of reducing the amount of time 
each light source is used (because of the time- 
multiplexing procedure), thus reducing the signal- 
to-noise ratio of the measurement. In the future, 
development of machines with a larger number of 
detectors may address this problem. The peak 
power of each diode is between 5 and 10 mW 
However, since the diodes are time-multiplexed, 
on average each source provides only between .3 
and .6 mW 

In this approach, a particular source-detector 
combination is referred to as a channel. Currently, 
in order to provide enough recording time for 
each channel and maintain a sufficient sampling 
rate (40-100 Hz), we can record data from 256 
channels during an optical recording session. 
However, to obtain sufficient spatial sampling and 
concurrently cover most of the scalp, we require 
something on the order of 500-1,000 channels. 
For this reason, we combine data from different 
(typically four) recording sessions (for a total of 
1,024 channels). In the future, the increase in the 
number of detectors may also help address this 
issue. 

To carry the light from the source to the head 
and from the head to the detectors, we use optic 
fibers. Those for the sources can be quite small 
(0.4 mm diameter); however, it is convenient to 
have large fibers for the detectors (3 mm diameter) 
to collect more light from the head. The fibers are 
held on the head using appropriately modified mo- 
torcycle helmets of different sizes, depending on 
the subject's head size. If the interest is in compar- 
ing different wavelengths, source fibers connected 
to different diodes are paired together. Prior to in- 
serting the detector fibers into the helmet, hair is 
combed away from the relevant locations using 
smooth instruments; the sources are so thin that 
this process is not necessary. 

Because the Imagent is an FD device, the 
sources are modulated at radio frequencies (typi- 
cally 100-300 MHz). A heterodyning frequency 
(typically 5-10 kHz) is used to derive signals at a 



much lower frequency than that at which the 
sources are modulated. This is obtained by feeding 
the PMTs with a slightly different frequency than 
the source (for example, 100.005 MHz instead of 
100 MHz). The use of a heterodyning frequency 
greatly reduces the cost of the machine. 

Frequency domain methods provide three pa- 
rameters for each data point and channel: (a) DC 
amplitude (average amount of light reaching the 
detector); (b) AC amplitude (oscillation at the 
cross-correlation frequency); and (c) phase delay 
(an estimate of photons' time of flight). AC or DC 
measures can be used to estimate the slow hemo- 
dynamic effects (spectroscopic approach), whereas 
phase delay measures can be used to estimate the 
fast neuronal effects (although intensity measures 
can also be used for this purpose). Using AC oscil- 
lations makes the measurements quite insensitive 
to environmental noise sources. Note that for de- 
vices based on CW methods, only one parameter is 
obtained (light intensity, equivalent to the DC am- 
plitude). 

Given the constraints imposed by our current 
recording apparatus, an important consideration 
with respect to the measurement of optical imaging 
data is to decide where on the head to place the 
source and detectors fibers. We refer to a particular 
set of sources and detector fibers as a montage. The 
design of an appropriate montage needs to take 
several considerations into account. First, it is cru- 
cial to avoid cross talk of different sources that can 
be picked up by the same detector. This implies 
that the light arriving at a particular detector at a 
particular time needs to come from only one 
source. This is facilitated by the fact that the prop- 
erties of the head make it very unlikely for photons 
to reach detectors that are located more than 
7-8 cm distance from a particular source. Another 
aspect of the design is that it is useful to maximize 
the number of channels in which the distance be- 
tween the source and detector is between 2 and 
5 cm. This range is determined by the fact that (a) 
when the source-detector distance is less than 
2 cm, most of the photons traveling between the 
source and the detector will not penetrate deep 
enough to reach the cortex; and (b) when the 
source-detector distance is greater than 5 cm, too 
few photons will reach the detector for stable 
measurements. Figure 5.6 shows an example of a 
montage covering the frontal cortex, as well as a 
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map of the projections of each channel color coded 
based on the source-detector distance. 

Digitization and Coregistration 

Coregistration is the procedure used to align the 
optical data with the anatomy of the brain. This 
step is necessary to relate the optical measurements 
to specific brain structures. Note that, as men- 
tioned earlier, EROS has a spatial resolution on the 
order of 5-10 mm. This means that using scalp 
landmarks alone (such as those exploited by the 
10/20 system used for EEG recordings) cannot be 
sufficient for appropriate representation of the 
data. This is because the relationship between scalp 
landmarks and underlying brain structures is very 
approximate (typically varying by several centime- 
ters between one person and another). Thus, in or- 
der to exploit the spatial resolution afforded by the 
EROS, we need a technology capable of achieving a 
subcentimeter level of coregistration with individ- 
ual brains' anatomy. This is particularly important 
when optical imaging is used with older adults, as 
individual differences in anatomy will increase 
with age. 

We have developed a procedure that reduces 
the coregistration error to 3-8 mm (Whalen, Fabi- 
ani, & Gratton, 2006). This procedure is based on 
the digitization of the source and detector locations 
(using a magnetic-field 3-D digitizer), as well as of 
three fiducials and a number of other points on the 
surface of the head. Using an iterative least square 
fitting procedure (Marquardt-Levenberg), the loca- 
tions of each individual subject are then fitted to 
the Tl MR images for the same subjects (in which 
the fiducial locations are marked with Beekley 
spots; see figure 5.6). The first step of the proce- 
dure involves fitting the fiducials, with the follow- 
ing steps fitting the rest of the points on the head 
surface. The locations for each source and detector 
are then transformed into Talairach coordinates, 
separately for each subject. 

Analysis 

In our laboratory, the analysis of optical data is 
conducted in two steps. The first step involves 
preprocessing of the optical data and is conducted 
separately for each channel. This involves: (a) cor- 
rection of phase wrapping around the 360-degree 



value; (b) polynomial detrending of slow drifts 
(less than .01 Hz); (c) correction of pulse artifacts 
(see Gratton & Corballis, 1995); (d) band-pass 
frequency filtering (if needed; Maclin et al, 2003); 
(e) segmentation into epochs; (f) averaging; and 
(g) baseline correction. The second step involves 
(a) 2-D or 3-D reconstruction of the data from each 
channel, (b) combining together data from differ- 
ent channels according to their locations on the sur- 
face of the head, and (c) statistical analyses across 
subjects using predetermined contrasts (following 
the same model used for analysis of fMRI and PET 
data — statistical parametric mapping [SPM]; Fris- 
ton et al., 1995). These analyses can be carried out 
independently for each data point in space or time. 
In many cases, a greater power is obtained by 
restricting the analysis (in both time and space) us- 
ing regions and intervals of interest (ROIs and 
IOIs). The ROIs and IOIs can be derived from 
the literature or from the fMRI and electrophysio- 
logical data obtained with the same paradigms and 
subjects. 



Conclusion and Considerations 
for Neuroergonomics 

Noninvasive optical imaging is a set of new tools 
for studying human brain function. Two main 
types of phenomena can be studied: (a) neuronal 
activity, occurring immediately after the presenta- 
tion of the stimuli or in preparation for responses; 
and (b) hemodynamic phenomena occurring with 
a few seconds' delay. These two phenomena can be 
measured together. In addition, optical imaging is 
compatible with other techniques, such as ERPs 
and fMRI. Since the recording systems are poten- 
tially portable, optical imaging methods can in 
principle be used in a number of applied settings 
to study brain function. Their main limitation is 
the penetration of the signal (a few centimeters 
from the surface of the head), which precludes 
the measurement of deep structures (at least in 
adults). 

Fast signals appear particularly attractive for 
their ability to study the time course of brain activ- 
ity coupled with spatial localization. In principle, 
this can be used to construct dynamic models of 
the brain in action, which would make it feasible 
to identify processing bottlenecks — possibly at the 
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individual subject level. For these research pur- 
poses, fast optical signals can be applied immedi- 
ately in experimental settings. However, more 
work is needed before they can be employed in 
applied settings. First, the current recording sys- 
tems are relatively cumbersome, requiring multi- 
ple sessions (and a laborious setup) to obtain the 
desired spatial sampling. This problem may be ad- 
dressed through machine design (e.g., by increas- 
ing the number of detectors and making the setup 
easier). Also, selection of target cortical regions 
may make the setup procedure much simpler. Sec- 
ond, the signal-to-noise ratio of the measures is 
relatively low. This problem may also be addressed 
(at least in part) through appropriate system de- 
sign (including the use of more detectors, pi de- 
tectors, and higher-modulation frequencies) as 
well as analytical methods (including appropriate 
filtering methods). 

Slow signals can be practically useful because 
of their ease of recording and good signal-to- 
noise ratio. Some laboratories have investigated the 
possibility of building machines combining good 
recording capabilities with very low weight, so that 
they can be mounted directly on the head of a sub- 
ject and be operated remotely through telemetry. 
Even with current technology, it is possible to 
record useful data with very compact machines 
mounted on a chair or that the subject can carry in 
a backpack. This would make it possible to per- 
form hemodynamic imaging in a number of ap- 
plied settings — something of clear importance for 
neuroergonomics. If it were possible to use similar 
devices also for the recording of fast optical signals, 
than the possibility of obtaining fast and localized 
optical measures of brain activity in applied set- 
tings would open new and exciting possibilities in 
neuroergonomics. 

In a series of studies, we have investigated the 
use of EROS to study brain preparatory processes 
(Agran, Low, Leaver, Fabiani, & Gratton, 2005; 
Leaver, Low, Jackson, Fabiani, & Gratton, 2005). 
The basic rationale of this work is that information 
about how the brain prepares for upcoming stimuli 
may be useful for detecting appropriate preparatory 
states during real-life operations, and that this in- 
formation may be used for designing appropriate 
human-machine interfaces that would adapt to 
changing human cognitive requirements. Specifi- 
cally, we have focused on the question of whether 
general preparation for information processing can 



be separated from specific preparation for particular 
types of stimuli and responses. Preparatory states 
were investigated using cueing paradigms in which 
cues, presented some time in advance of an impera- 
tive stimulus, predicted which particular stimulus 
or response dimension was to be used on an up- 
coming trial. Preliminary data indicate that EROS 
activity occurring between the cue and the impera- 
tive stimulus signaled the occurrence of both gen- 
eral preparatory states and specific preparatory 
states for particular stimulus or response dimen- 
sions. If this type of measurement could be applied 
to real-life situations online, it could provide useful 
information for a neurocognitive human-machine 
interface. 



Main Points 

1 . Noninvasive optical imaging possesses several 
key features that make it a very useful tool for 
studying brain function, including: 

a. Concurrent sensitivity to both neuronal and 
hemodynamic phenomena 

b. Good combination of spatial and temporal 
resolution 

c. Adaptability to a number of different 
environments and experimental conditions 

d. Relatively low cost (when compared to 
MEG, PET, and fMRI) 

2. Limitations of this approach include: 

a. Limited tissue penetration (up to 5 cm or so 
from the surface of the adult head) 

b. Relatively low signal-to-noise ratio (for the 
fast neuronal signal — the slow 
hemodynamic signal is large and can be 
potentially observed on single trials) 

3. The characteristics of optical methods, and of 
fast signals in particular, make them promising 
methods in neuroimaging, especially for 
mapping the time course of brain activity over 
the head and its relationship with 
hemodynamic responses. 
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Transcranial Doppler Sonography 



Doppler ultrasound has been used in medicine for 
many years. The primary long-standing applica- 
tions include monitoring of the fetal heart rate dur- 
ing labor and delivery and evaluating blood flow in 
the carotid arteries. Applications developed largely 
in the last two decades have extended its use to vir- 
tually all medical specialties (Aaslid, Markwalder, 
& Nornes, 1982; Babikian & Wechsler, 1999; 
Harders, 1986; Risberg, 1986; Santalucia & Feld- 
mann, 1999). More recently Doppler ultrasound 
has been employed in the field of psychology to 
measure the dynamic changes in cerebral blood 
perfusion that occur during the performance of a 
wide variety of mental tasks. That application is the 
focus of this chapter. 

The close coupling of cerebral blood flow with 
cerebral metabolism and neural activation (Deppe, 
Ringelstein, & Knecht, 2004; Fox & Raichle, 1986; 
Fox, Raichle, Mintun, & Dence, 1988) provides in- 
vestigators with a mechanism by which to study 
brain systems and cognition via regional cerebral 
blood flow. Local distribution patterns of blood 
perfusion within the brain can be evaluated with 
high spatial resolution using invasive neuroimag- 
ing techniques like positron emission tomography 
(PET), functional magnetic resonance imaging 
(fMRI), and single photon emission computed to- 



mography (SPECT; see Fox & Raichle, 1986; Kan- 
del, Schwartz, & Jessel, 2000; Raichle, 1998). 
However, the high cost of these techniques cou- 
pled with a need for the injection of radioactive 
material into the bloodstream and the requirement 
that observers remain relatively still during testing 
limits their application in measuring the intracra- 
nial blood flow parameters that accompany neu- 
ronal activation during the performance of mental 
tasks, especially over long periods of time (Duschek 
& Schandry, 2003). One brain imaging supple- 
ment or surrogate for the field of neuroergonomics 
is transcranial Doppler sonography (TCD). Al- 
though its ability to provide detailed information 
about specific brain loci is limited in comparison 
to the other imaging procedures, TCD offers good 
temporal resolution and, compared to the other 
procedures, it can track rapid changes in blood 
flow dynamics that can be related to functional 
changes in brain activity in near-real time under 
less restrictive and invasive conditions (Aaslid, 
1986). This chapter summarizes the technical and 
anatomical elements of ultrasound measurement, 
the important methodological issues to be consid- 
ered in its use, and the general findings regarding 
TCD-measured hemovelocity changes during the 
performance of mental tasks. 



82 



Transcranial Doppler Sonography 



83 



Basic Principles of Ultrasound 
Doppler Sonography 

Doppler Fundamentals 

The transcranial Doppler method, first described 
by Aaslid et al. (1982), enables the continuous 
noninvasive measurement of blood flow velocities 
within the cerebral arteries. The cornerstone of the 
transcranial Doppler technique can be traced back 
to the Austrian physicist Christian Doppler, who 
developed the principle known as the Doppler ef- 
fect in 1843. The essence of this effect is that the 
frequency of light and sound waves is altered if the 
source and the receiver are in motion relative to 
one another. Transcranial Doppler employs ultra- 
sound from its source or transducer, which is di- 
rected toward an artery within the brain. The shift 
in frequency occurs when ultrasound waves or sig- 
nals are reflected by erythrocytes (red blood cells) 
moving through a blood vessel. As described by 
Harders (1986), the Doppler shift is expressed by 
the formula: 



F = 2(F x V x cos oc)/C 



(1) 



where F = the frequency of the transmitted ultra- 
sound, V = the real blood flow velocity, a = the an- 
gle between the transmitted sound beam and the 
direction of blood flow, and C = the velocity of the 
sound in the tissue (1,550 m/s in the soft tissue). 
The magnitude of the frequency shift is directly 
proportional to the velocity of the blood flow 
(Duschek & Schandry, 2003). 



Blood vessels within the brain that are exam- 
ined routinely with TCD include the middle cere- 
bral artery (MCA), the anterior cerebral artery 
(ACA), and the posterior cerebral artery (PCA). The 
MCA carries 80% of the blood flow within each ce- 
rebral hemisphere (Toole, 1984). Hemovelocity in- 
formation from this artery provides a global index 
within each hemisphere of blood flow changes that 
accompany neuroactivation. More region-specific 
information regarding hemovelocity and neuronal 
activity can be obtained from the ACA, which feeds 
blood to the frontal lobes and to medial regions of 
the brain, and from the PCA, which supplies blood 
to the visual cortex. The locations of these arteries 
within the brain are illustrated in figure 6.1. In gen- 
eral, blood flow velocities, measured in centimeters 
per second, are fastest in the MCA followed in turn 
by the ACA and the PCA (Sortberg, Langmoen, Lin- 
degaard, & Nornes, 1990). 

Dynamic adaptive blood delivery to neuronal 
processes, "neurovascular coupling," is controlled 
by the contraction and dilation of small cerebral 
vessels that result from the changing metabolic de- 
mands of the neurons. When an area of the brain 
becomes metabolically active, as in the performance 
of mental tasks, by-products of this activity, such as 
carbon dioxide, increase. This results in elevation of 
blood flow to the region to remove the waste prod- 
uct (Aaslid, 1986; Risberg, 1986). Therefore, TCD 
offers the possibility of measuring changes in meta- 
bolic resources during task performance (Stroobant 
& Vingerhoets, 2000). In addition to C0 2 changes, 
the cerebral vascular regulating mechanisms also 
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Figure 6.1. Measurement of cerebral 
blood flow in the middle cerebral 
artery (MCA). The anterior and poste- 
rior cerebral arteries (ACA and PCA, 
respectively) can be insonated as well. 
Reprinted from Neuroimage, 21, 
Deppe, M., Ringelstein, E. B., & 
Knecht, S., The investigation of func- 
tional brain lateralization by transcra- 
nial Doppler sonography, 1124-1146, 
copyright 2004, with permission from 
Elsevier. 



84 



Neuroergonomics Methods 



involve biochemical mediators that include potas- 
sium (+K), hydrogen (+H), lactate, and adenosine. 
More detailed descriptions of the biochemical me- 
diators can be found in Hamman and del Zoppo 
(1994) and Watanabe, Nobumasa, and Tadafumi 
(2002). One might assume that the increase in 
blood flow also serves to deliver needed glucose, 
but this possibility is open to question (Raichle, 
1989, 1998). It is important to note, as Duschek 
and Schandry (2003) have done, that the diameters 
of the larger arteries, the MCA, ACA, and PAC, re- 
main largely unchanged under varying task de- 
mands, indicating that hemovelocity changes in the 
large arteries do not result from their own vascular 
activity but instead from changes in the blood de- 
manded by their perfusion territories and thus, 
changes in local neuronal activity. 

TCD Instrumentation 

Two types of examination procedures are available 
with most modern transcranial Doppler devices — 
the continuous wave (CW) and pulsed wave (PW) 
procedures. The former utilizes ultrasound from a 
single crystal transducer that is transmitted contin- 
uously at a variety of possible frequencies to a tar- 
geted blood vessel while the returning signal 



(backscatter) is received by a second crystal. Mea- 
surement inaccuracies can exist with this procedure 
because it does not screen out signals from vessels 
other than the one to be specifically examined or 
insonated. The pulsed-wave transcranial Doppler 
procedure employs a single probe or transducer 
that is used both to transmit ultrasound waves at a 
frequency of 2 MHz and to receive the backscatter 
from that signal. Dopplers that employ pulsed ul- 
trasound allow the user to increase or decrease the 
depth of the ultrasound beam to isolate a single ves- 
sel and thus increase the accuracy of the data. Con- 
sequently, the pulsed Doppler is the one employed 
most often in behavioral studies. 

The TCD device uses a mathematical expression 
known as a fast Fourier transform to provide a pic- 
torial representation of blood flow velocities in real 
time on the TCD display as illustrated in figure 6.2 
(Deppe, Knecht, Lohmann, & Ringelstein, 2004). 
This information can then be stored in computer 
memory for later analysis. 

Modern Transcranial Doppler devices have the 
capacity to measure blood flow velocities simul- 
taneously in both cerebral hemispheres. When 
performing an ultrasound examination, the trans- 
ducer is positioned above one of three ultrasonic 
windows. Illustrated in figure 6.3 are the anterior 
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Figure 6.2. Transcranial Doppler display showing flow velocity waveforms, mean 
cerebral blood flow velocity, and depth of the ultrasound signal. See also color insert. 
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Figure 6.3. Subareas of the transcranial temporal window. (A): F, frontal; A, anterior; M, middle; and 
P, posterior. (B): Transducer angulations vary according to the transtemporal window utilized. Reprinted 
with permission from Aaslid (1986). 



temporal, middle temporal, and posterior temporal 
windows, located just above the zygomatic arch 
along the temporal bone (Aaslid, 1986). These ar- 
eas of the skull allow the ultrasound beam to pene- 
trate the cranium and enter into the brain, thereby 
allowing for the identification of blood flow activ- 
ity in the MCA, PCA, and ACA (Aaslid, 1986). The 
cerebral arteries can be identified by certain char- 
acteristics that are specific to each artery, including 
depth of insonation, the direction of blood flow, 
and blood flow velocity, as described in table 6.1 
(after Aaslid, 1986). 

In clinical uses, the patient is supine on the ex- 
amination table. Participants are seated in an upright 
position during behavioral testing. To begin the TCD 
examination, the power setting on the Doppler is in- 
creased to 100%, and the depth is set to 50 mm. The 

Table 6.1 . Criteria for Artery Identification 



Blood Vessel 



Direction of Mean 

Depth Flow in Relation Velocity 
(mm) to Transducer (cm/s) 



Middle cerebral 

artery 30—67 Toward 62 +/— 12 

Anterior cerebral 

artery 60-80 Away 50+/-11 

Posterior cerebral 

artery 55-70 Toward 39 +/-10 



transducer is prepared by applying ultrasonic gel to 
its surface. The gel serves as a coupling medium that 
allows the ultrasound to be transmitted from the 
transducer to the targeted blood vessel in the cra- 
nium. The transducer is then placed in front of the 
opening of the ear above the zygomatic arch and 
slowly moved toward the anterior window until the 
MCA is found. Once the MCA is located, the exami- 
nation of other arteries can be accomplished by ad- 
justing both the angle of the transducer and depth of 
the ultrasound beam. Due to the salience of the 
transducer, participants can accurately remember its 
position on the skull. Accordingly, that information 
can be reported to the investigator and utilized in a 
subsequent step in which the investigator mounts 
the transducer in a headband to be worn by the par- 
ticipant during testing. 

Headband devices can vary in design from an 
elastic strap and mounting bracket that holds a single 
transducer, as shown in figure 6.4 (left), to a more 
elaborate design that is fashioned after a welder's hel- 
met headband, illustrated in figure 6.4 (right). Addi- 
tionally, the headband may be designed with 
transducer mounting brackets on the left and right 
sides accommodating two transducers that are used 
in simultaneous bilateral blood flow measurement. 
This configuration is also shown in figure 6.4. (right). 
The low weight and small size of the transducer and 
the ability to insert it conveniently into a headband 
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Figure 6.4. Transducer mounting designs: (left) elastic headband mounting bracket configured with a single 
transducer for unilateral recording; (right) welder's helmet mounting bracket affording bilateral recording capability. 



mounting bracket permit real-time measurement of 
cerebral blood flow velocity while not limiting mo- 
bility or risking vulnerability to body motion. 

Applying the headband to the participant is the 
most critical step in the instrumentation process. 
The mounting brackets shown in figure 6.4 must 
be aligned with the ultrasonic window where the 
original signal was found; care must also be taken 
to secure the headband to the participants skull in 
a comfortable manner that is neither too tight nor 
too loose. Once the headband is in place, the trans- 
ducer is again prepared by applying ultrasonic gel 
and then inserted into the mounting bracket until 
it makes contact with the skull. At this point, posi- 
tioning adjustments are made to the transducer to 
reacquire the Doppler signal. As previously de- 
scribed, the MCA is typically used as the target ar- 
tery; the depth and angle of the transducer can 
then be altered to acquire the ACA and PCA, if de- 
sired (see figure 6.4). Upon reacquiring the signal, 
the transducer is secured in the mounting bracket 
using a tightening device that accompanies the 
mounting bracket system. After the transducer has 
been tightened, data collection can begin. 

Most studies measure task-related variations 
in blood flow as a percentage of a resting baseline 
value. Consequently, the acquisition of baseline 
blood flow velocity data is a key element in TCD 
research. Several strategies are possible for deter- 
mining baseline values. The simplest is to have par- 
ticipants maintain a period of relaxed wakefulness 
for an interval of time, e.g., 5 minutes, and use he- 
movelocity during the final 60 seconds of the base- 
line period as the index against which to compare 
blood flow during or following task performance 



(Aaslid, 1986; Hitchcock et al., 2003). Such a strat- 
egy is most useful when only a single mental task 
is involved. Strategies become more complex when 
multiple tasks are employed. As described by 
Stroobant and Vingerhoets (2000), one approach in 
the multiple-task situation is to use an alternating 
rest-activation strategy in which a rest condition 
precedes each task. Each rest phase can then be 
used as the baseline measurement for its subse- 
quent mental task, or the average of all rest condi- 
tions can be employed to determine blood flow 
changes in each task. An advantage of the first op- 
tion is the likelihood that artifactual physiological 
changes between measurements will be minimized 
because the resting and testing phases occur in 
close temporal proximity. The second option offers 
the advantage of increased reliability in the baseline 
index, since it is based upon a larger sample of data. 
Stroobant and Vingerhoets (2000) noted that an al- 
ternate strategy under multiple-task conditions is to 
make use of successive cycles of rest and activation 
within tasks. They pointed out that while the tem- 
poral proximity between rest and activation is high 
with this technique, care must be taken to ensure 
that task-induced activation has subsided before the 
beginning of a rest phase is defined. 

Additional issues to be considered in develop- 
ing baseline measures are (1) whether participants 
should have their eyes open or closed during the 
baseline phase (Berman & Weinberger, 1990) and 
(2) whether they should sit calmly, simply looking 
at a blank screen (Bulla-Hellwig, Vollmer, Gotzen, 
Skreczek, & Hartje, 1996), or watching the dis- 
play of the criterion task unfold on the screen with- 
out a work imperative (Hitchcock et al., 2003). At 
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present, there is no universal standard for baseline 
measurement in TCD research. As Stroobant and 
Vingerhoets (2000) suggested, future studies are 
needed to compare the different methodological 
approaches to this crucial measurement issue. 

Additional Methodological Concerns 

As with any physiological measure, certain exclu- 
sion criteria apply in the use of the TCD procedure. 
In a careful review of this issue, Stroobant and 
Vingerhoets (2000) pointed out that participants in 
TCD investigations are frequently required to ab- 
stain from caffeine, nicotine, and medication. The 
duration of abstention has varied widely across 
studies, with times ranging from 1 to 24 hours prior 
to taking part in the investigation. Stroobant and 
Vingerhoets (2000) recommended that the absten- 
tion period be at least 12 hours prior to the study. 

Due to concerns about handedness and lateral- 
ized functions that are prevalent in the neuro- 
science literature (Gazzaniga, Ivry, & Mangun, 
2002; Gur et al., 1982; Markus & Boland, 1992; 
Purves et al., 2001), investigators employing the 
TCD procedure tend to restrict their participant 
samples to right-handed individuals. Moreover, 
since age is known to influence cerebral blood 
flow, with lower levels of blood flow at rest and 
during task performance in older people (Droste, 
Harders, & Rastogi, 1989b; Orlandi & Murri, 
1996), and since there is some evidence pointing 
to higher blood flow velocities in females than 
males of similar age (Arnolds & Von Reutern, 
1986; Rihs et al, 1995; Vingerhoets & Stroobandt, 
1999; Walter, Roberts, & Brownlow, 2000), it is 
important to control for age and gender effects in 
TCD studies. Evidence is also available to show 
that blood flow is sensitive to emotional processes 
(Gur, Gur, & Skolnick, 1988; Troisi et al., 1999; 
Stoll, Hamann, Mangold, Huf, & Winterhoff- 
Spurk, 1999), leading Stroobant and Vingerhoets 
(2000) to suggest that investigators should try to 
reduce participants' levels of anxiety in TCD stud- 
ies. Finally, since variations in cerebral hemody- 
namics during task performance have been found 
to differ between controls and patients suffer- 
ing from migraine headache, cerebral ischemia, 
and cardiovascular disease (Backer et al., 1999; 
Stroobant, Van Nooten, & Vingerhoets, 2004), par- 
ticipants' general health is also of concern in TCD 
investigations. 



Reliability and Validity 

As with any measurement procedure, the potential 
usefulness of the TCD technique will rest upon its 
reliability and validity. Since the baseline values com- 
prise the standard from which task-determined ef- 
fects are derived, one would expect the Doppler 
technique to yield baseline data that are reproducible 
over time and independent of spontaneous autoreg- 
ulatory fluctuations in the brain. Several investiga- 
tions have reported a reasonably satisfactory range 
of baseline reliability coefficients between .71 and 
.98 (Baumgartner, Mathis, Sturzenegger, & Mattle, 
1994; Bay-Hansen, Raven, & Knudsen, 1997; 
Maeda et al, 1990; Matthews, Warm, & Washburn, 
2004; Totaro, Marini, Cannarsa, & Prencipe, 1992). 
In addition, Matthews et al. (2004) and Schmidt 
et al. (2003) have also reported significant inter- 
hemispheric correlations for resting baseline values. 
Results such as these indicate that individual differ- 
ences in baseline values are highly consistent. In ad- 
dition to baseline reliability, other studies have 
demonstrated strong test-retest reliabilities in task- 
induced blood flow changes. Knecht et al. (1998) re- 
ported a reliability coefficient of .90 for blood flow 
responses obtained from two consecutive examina- 
tions on a word-fluency task; Matthews et al. (2004) 
reported significant intertask correlations ranging 
between .42 and .66 between blood flow responses 
induced by a battery of high-workload tasks includ- 
ing line discrimination, working memory, and track- 
ing, and Stroobant and Vingerhoets (2001) reported 
test-retest reliability values ranging from .61 to .83 
for eight different verbal and visuospatial tasks. 

Partial convergent validity for the TCD pro- 
cedure comes from studies showing that factors 
known to decrease cerebral blood volume, such as 
when observers are passively exposed to increased 
gravitational force, also reduce TCD measured 
blood flow (Tripp & Chelette, 1991). More defini- 
tive convergent validity has been determined for 
the TCD technique by comparing task-induced 
blood flow changes obtained with this procedure 
to those secured by PET, fMRI, and the WADA test 
(Wada & Rasmussen, 1960), a procedure often 
used in identifying language lateralization. Similar 
results have been obtained with these techniques 
(Duschek & Schandry, 2003; Jansen et al., 2004; 
Schmidt et al., 1999), and substantial correlations, 
some as high as .80, have been reported (Deppe 
et al., 2000; Knake et al., 2003; Knecht et al., 1998; 
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Rihs, Sturzenegger, Gutbrod, Schroth, & Mattle, 
1999). Further validation of the TCD procedure 
comes from studies showing that changes in he- 
movelocity during the performance of complex 
mental tasks that are noted with this procedure are 
not evident when observers simply gaze at stimu- 
lus displays without a work imperative (Hitchcock 
et al, 2003) and that such changes are not related 
to more peripheral reactions involving changes in 
breathing rate, heart rate, blood pressure, or end- 
tidal carbon dioxide (Kelley et al., 1992; Klingel- 
hofer et al, 1997; Schnittger, Johannes, & Munte, 
1996;Stolletal., 1999). 



TCD and Psychological Research 

Roots in the Past 

The possibility that examining changes in brain 
blood flow during mental activities could lead to 
an understanding of the functional organization of 
the brain was recognized in the nineteenth century 
by Sir Charles Sherrington (Roy & Sherrington, 
1890) and discussed by William James (1890) in 
his classic text Principles of Psychology. The first em- 
pirical observation linking brain activity and cere- 
bral blood flow was made in a clinical study by 
John Fulton (1928), who reported changes in blood 
flow during reading in a patient with a bony defect 
over the primary visual cortex. Since those early 
years, research with the PET and fMRl procedures 
has provided considerable evidence for a close tie 
between cerebral blood flow and neural activity 
during the performance of mental tasks (Raichle, 
1998; Risberg, 1986). 

Work with the fTDC procedure has added sub- 
stantially to this evidence by showing that changes 
in blood flow velocity occur in a wide variety of 
tasks varying from simple signal detection to com- 
plex information processing. The literature is exten- 
sive and space limitations permit us to only describe 
some of these findings. More extensive coverage 
can be found in reviews by Duschek and Schandry 
(2003), Klingelhofer, Sander, and Wittich (1999), 
and Stroobant and Vingerhoets (2001). 

Basic Perceptual Effects 

The presentation of even simple visual stimuli can 
cause variations in flow velocity in the PCA. Studies 



have shown that visual signals are accompanied by 
increases in blood flow in this artery (Aaslid, 1987; 
Conrad & Klingelhofer, 1989, 1990; Kessler, Bohn- 
ing, Spelsperg, & Kompf, 1993), that these he- 
movelocity changes will occur to stimuli as brief as 
50 ms in duration, and that the magnitude of the 
hemovelocity increase is directly related to the in- 
tensity of the stimuli and to the size of the visual 
field employed (Sturzenegger, Newell, & Aaslid, 
1996; Wittich, Klingelhofer, Matzander, & Conrad, 
1992). Hemovelocity increments tend to be greater 
to intermittent than to continuous light (Conrad & 
Klingelhofer, 1989, 1990; Gomez, Gomez, & Hall, 
1990), and there is evidence that flow velocity in- 
crements might be lateralized to the right hemi- 
sphere in response to blue, yellow, and red light 
(Njemanze, Gomez, & Horenstein, 1992). It is im- 
portant to note that the flow velocities in the PCA 
accompanying visual stimulation are greater when 
observers are required to search the visual field for 
prespecified targets or to recognize symbols than 
when they simply look casually at the displays 
(Schnittger et al., 1996; Wittich et al, 1992). The 
fact that the dynamics of the blood flow response 
are affected by how intently observers view the vi- 
sual display indicates that more than just the physi- 
cal character of the stimulus determines the nature 
of the blood flow response. As Duschek and 
Shandry (2003) pointed out, the Doppler record- 
ings of blood flow changes in the PCA clearly reflect 
visually evoked activation processes. 

Changes in cerebral blood flow are not limited 
to visual stimuli, however; they occur to acoustic 
stimuli as well. Klingelhofer and his associates 
(1997) have shown a bilateral increase in the MCA 
to a white noise signal and an even larger hemove- 
locity increase when observers listened to samples 
of speech. In the latter case, the blood flow changes 
were lateralized to the left MCA, as might be antic- 
ipated from left-hemispheric dominance in the 
processing of auditory language (Hellige, 1993). In 
other studies, observers listened to musical pas- 
sages in either a passive mode or under conditions 
in which they had to identify or recognize the 
melodies that they heard. Passive listening led to 
bilateral increases in blood flow in the MCA, while 
in the active mode, blood flow increments were lat- 
eralized to the right hemisphere (Matteis, Silvestrini, 
Troisi, Cupini, & Caltagirone, 1997; Vollmer-Haase, 
Finke, Hartje, & Bulla-Hellwig, 1998). The latter 
result coincides with PET studies indicating right 
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hemisphere dominance in the perception of melody 
(Halpern & Zatorre, 1999; Zatorre, Evans, & Meyer, 
1994). 

In still another experiment involving acoustic 
stimulation, Vingerhoets and Luppens (2001) ex- 
amined blood flow dynamics in conjunction with 
a number of dichotic listening tasks that varied 
in difficulty. Blood flow responses in the MCAs of 
both hemispheres varied directly with task diffi- 
culty, though with a greater increase in the right 
hemisphere. However, unlike prior results with the 
PET technique in which directing attention to 
stimuli in one ear led to more pronounced blood 
flow in the contralateral hemisphere during di- 
chotic listening (O'Leary et al., 1996), blood flow 
asymmetries were not noted in this study. The 
authors suggested that hemodynamic changes 
caused by attentional strategies in the dichotic 
listening task may be too subtle to be detected as 
lateralized changes in blood flow velocity. Never- 
theless, the more pronounced right hemispheric 
response to the attention tasks employed in this 
study is consistent with the dominant role of the 
right hemisphere in attention described in PET 
studies (Pardo, Fox, & Raichle, 1991) and in 
other TCD experiments involving short-term atten- 
tion (Droste, Harders, & Rastogi, 1989a; Knecht, 
Deppe, Backer, Ringelstein, & Henningsen, 1997) 
and vigilance or long-term attention tasks (Hitch- 
cock et al., 2003). Transcranial Doppler research 
with vigilance tasks is described more fully in 
chapter 10. 

Hemovelocity changes occur not only to the 
presentation of stimuli but also to their antici- 
pation. Along this line, Backer, Deppe, Zunker, 
Henningsen, and Knecht (1994) performed an ex- 
periment in which observers were led to expect 
threshold-level tactual stimulation on the tip of ei- 
ther their right or left index finger 5 seconds after 
receiving an acoustic cue. Blood flow in the MCA 
contralateral to the side of the body in which stim- 
ulation was expected increased about 3 seconds 
prior to the stimulus, indicating an attention- 
determined increase in cortical activity. These find- 
ings were supported in a similar study by Knecht et 
al. (1997) in which blood flow in the contralateral 
MCA also increased following the acoustic cue, but 
only when subsequent stimulation was anticipated. 
No changes were observed in conditions in which 
observers were presented with tactual stimulation 
alone, indicating higher cortical activation in con- 



nection with an anticipation of stimulation than for 
activation associated with the tactual stimulus it- 
self. In still another experiment, this group (Backer 
et al, 1999) demonstrated that the expectancy ef- 
fect was tied to stimulus intensity. Using near- 
threshold tactual stimuli, they again found more 
pronounced blood flow in the MCA contralateral 
to the side where the stimulus was expected. In 
contrast, anticipation for stimuli well above thresh- 
old showed right-dominant blood flow increases 
that were independent of the side of stimulation. 
The finding of anticipatory blood flow with the 
TCD procedure is similar to that in PET studies 
showing increased blood flow in the postcentral 
cortex during the anticipation of tactual stimuli 
(Roland, 1981). 

Complex Information Processing 

Studies requiring the relatively simple detection 
and recognition of stimuli have been accompanied 
by experiments that focused on more complex 
tasks. As a case in point, we might consider a series 
of studies by Droste, Harders, and Rastogi (1989a, 
1989b) in which recordings were made of blood 
flow velocities in the left and right MCAs during 
the performance of six different tasks: reading 
aloud, dot distance estimation, noun finding, spa- 
tial perception, multiplication, and face recogni- 
tion. All of the tasks were associated with increases 
in blood flow velocity, with the largest increase oc- 
curring when observers were required to read ab- 
stract four-syllable nouns aloud. In a related study, 
this group of investigators measured hemovelocity 
changes bilaterally while observers performed 
tasks involving spatial imaging and face recogni- 
tion. Increments in blood flow velocity were also 
noted in this study (Harders, Laborde, Droste, & 
Rastogi, 1989a). 

The findings described above regarding the 
ability of linguistic tasks to increase blood flow in 
the MCA have been reported in several other inves- 
tigations, which essentially revealed left hemi- 
spheric dominance in right-handed observers and 
higher variability in left-handers (Bulla-Hellwig 
et al., 1996; Markus & Boland, 1992; Njemanze, 
1991; Rihs et al., 1995; Varnadore, Roberts, & 
McKinney, 1997; Vingerhoets & Stroobant, 1999). 
Moreover, Doppler-measured blood flow incre- 
ments while solving arithmetic problems have also 
been described by Kelley et al. (1992) and Thomas 
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and Harer (1993). Similarly, hemovelocity incre- 
ments have also been described when observers are 
engaged in a variety of spatial processing tasks. 
Vingerhoets and Stroobant (1999) have reported 
right hemispheric specialization in a task involving 
the mental rotation of symbols, while in a study in- 
volving the mental rotation of cubes, Serrati et al. 
(2000) reported that blood flow was directly re- 
lated to the difficulty of the rotation task. Other 
spatial tasks in which blood flow increments were 
obtained are compensatory tracking (Zinni & Para- 
suraman, 2004) and the Multi-Attribute Task Bat- 
tery (Comstock & Arnegard, 1992), a mah-jongg 
tile sorting task Kelly et al. (1993), and a simulated 
flying task (Wilson, Finomore, & Estepp, 2003). In 
the latter case, blood flow was found to vary di- 
rectly with task difficulty. Along with blood flow 
changes occurring during the execution of com- 
plex performance tasks, studies by Schuepbach 
and his associates have reported increments in 
blood flow velocity in the MCA and the ACA that 
were linked to problem difficulty on the Tower of 
Hanoi and Stockings of Cambridge Tests, which 
measure executive functioning and planning ability 
(Frauenfelder, Schuepbach, Baumgartner, & Hell, 
2004; Schuepbach et al, 2002). Clearly, in addi- 
tion to detecting rapid blood flow changes in sim- 
ple target detection or recognition tasks, the TCD 
procedure is also capable of detecting such changes 
during complex cognitive processing, and in some 
cases increments in cerebral blood flow have been 
shown to vary directly with task difficulty (Frauen- 
felder et al., 2004; Schuepbach et al., 2002; Serrati 
et al., 2000; Wilson et al., 2003). These studies in- 
volving variations in difficulty are critical because 
they demonstrate that the linkage between blood 
flow and complex cognitive processing is more 
than superficial. It is worth noting, however, that 
the studies that have established the blood flow- 
difficulty linkage have done so using only two 
levels of difficulty. To establish the limits of the 
blood flow-difficulty linkage, studies are needed 
that manipulate task difficulty in a parametric 



Conclusions and Neuroergonomic 
Implications 

The research reviewed in this chapter indicates that 
fTDC is an imaging tool that allows for fast and 



mobile assessment of task-related brain activation. 
Accordingly, TCD could be used to assess task dif- 
ficulty and task engagement, to evaluate hemi- 
spheric asymmetries in perceptual representation, 
and perhaps to determine when operators are in 
need of rest or replacement. It may also be useful 
in determining when operators might benefit from 
adaptive automation, an area of emerging interest 
in the human factors field in which the allocation 
of function between human operators and com- 
puter systems is flexible and adaptive assistance is 
provided when participants are in need of it due to 
increased workload and fatigue (Parasuraman, 
Mouloua, & Hilburn, 1999; Scerbo, 1996; Wick- 
ens & Hollands, 2000; Wickens, Mavor, Parasura- 
man, & McGee, 1998; chapter 16, this volume). In 
these ways, the TCD procedure could lead to a 
greater understanding of the interrelations between 
neuroscience, cognition, and action that are basic 
elements in the neuroergonomic approach to the 
design of technologies and work environments for 
safer and more efficient operation (Parasuraman, 
2003; Parasuraman & Hancock, 2004). 



Main Points 

1. Ultrasound can be used to measure brain 
blood flow velocity during the performance of 
mental tasks via the transcranial Doppler 
procedure. 

2. The transcranial Doppler procedure is a 
flexible, mobile, relatively inexpensive, and 
noninvasive technique to measure brain 
blood flow during the performance of mental 
tasks. 

3. Transcranial Doppler sonography offers the 
possibility of measuring changes in metabolic 
resources during task performance. 

4. Transcranial Doppler measurement has good 
temporal resolution, substantial reliability, and 
convergent validity with other brain imaging 
procedures such as PET and fMRI. 

5. Transcranial Doppler measurement has been 
successful in linking brain blood flow to per- 
formance in a wide variety of tasks ranging 
from simple signal detection to complex 
information processing and in identifying cere- 
bral laterality effects. 

6. From a neuroergonomic perspective, 
transcranial Doppler measurement may be 
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useful in determining when operators are task 
engaged and when they are in need of rest or 
replacement. 
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Eye Movements as a Window 
on Perception and Cognition 



Eye tracking has been a tool in human factors since 
the first days of the field, when Fitts, Jones, and 
Milton (1950) analyzed the eye movements of pi- 
lots flying instrument landing approaches. Lacking 
computerized eye tracking equipment, the re- 
searchers recorded data by filming their subjects 
with a movie camera mounted in the cockpit. Once 
collected, the data, thousands of frames, were 
coded by hand. The results gave insight into the pi- 
lots' scanning habits and revealed shortcomings in 
cockpit display design that might have gone undis- 
covered through any other methodology. The fre- 
quency with which pilots looked at a given 
instrument, for example, was determined by the im- 
portance of that instrument to the maneuver being 
flown; the duration of each look was determined by 
the difficulty of interpreting the instrument; pat- 
terns of scanning varied across individuals; and 
instruments were arranged in a pattern that en- 
couraged a suboptimal pattern of transitions be- 
tween channels. Such information, painstakingly 
acquired, provided an important foundation for 
the understanding of expert and novice pilot per- 
formance and pointed directly toward potential 
improvements in cockpit design. 

Happily, sophisticated eye tracking systems are 
now abundant, inexpensive, easy to use, and often 



even portable. Oculomotor data are thus far easier 
to collect and analyze than they were at the time of 
Fitts's work (see Jacob & Karn, 2003, for discus- 
sion of developments in eye tracking technology 
and the history of eye tracking in engineering psy- 
chology). The cognitive and neural mechanisms 
that control eye movements, moreover, have be- 
come better understood with the advent of cogni- 
tive neuroscience, putting researchers in a stronger 
position to predict, control, and interpret oculo- 
motor behavior. This is compatible with the neu- 
roergonomics view (Parasuraman, 2003) that an 
understanding of the brain mechanisms underlying 
eye movements can enhance applications of this 
measure to investigate human factors issues. Ac- 
cordingly, eye movement data are being used by 
ergonomics researchers in an ever-growing vari- 
ety of domains — including radiological diagnosis 
(e.g., Carmody Nodine, & Kundel, 1980; Kundel & 
LaFollette, 1972), driving (e.g., Mourant & Rock- 
well, 1972), reading (e.g., McConkie & Rayner, 
1975; Rayner, 1998), airport baggage screening (Mc- 
Carley, Kramer, Wickens, Vidoni, & Boot, 2004), 
and athletic performance (Abernethy, 1988) — and 
for a range of purposes. They are often studied for 
understanding of the perceptual-cognitive processes 
and strategies mediating performance in complex 
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tasks. This may include identifying nonoptimal 
aspects of a human-machine interface (e.g., Fitts 
et al., 1950), delineating expert-novice differences 
in the performance of a given task (e.g., Kundel & 
LaFollette, 1972), or revealing the nature of the 
lapses that produce performance errors (e.g., Mc- 
Carley, Kramer, et al., 2004). Such knowledge can 
inform future human factors interventions such as 
the redesign of visual displays or the development 
of training programs. Oculomotor data can also be 
used to draw inferences about an operator's cogni- 
tive state or mental workload level (May, Kennedy, 
Williams, Dunlap, & Brannan, 1990; Wickens & 
Hollands, 2000) and may even be diagnostic of the 
nature of mental workload imposed by a nonvisual 
secondary task (Recarte & Nunes, 2000). Research 
on interface design, finally, has suggested that eye 
movements may be useful as a form of control in- 
put in human-computer interaction, providing a 
means for users to perform onscreen point-and- 
click operations (Surakka, Illi, & Isokoski, 2003), 
for example, or to control the display of visual 
information (Reingold, Loschky McConkie, & 
Stampe, 2003). 

In this chapter, we briefly review findings from 
the basic study of eye movements in cognitive psy- 
chology and perception and consider implications of 
these findings for human factors researchers and 
practitioners. Because of constraints on space, dis- 
cussion focuses largely on the aspects of eye move- 
ments that pertain to human performance in visual 
search and supervisory monitoring tasks or human 
interaction with machine systems. For a more com- 
prehensive review and discussion of the eye move- 
ment literature, including consideration of the role of 
eye movements in reading, see Findlay and Gilchrist 
(2003), Hyona, Radach, and Deubel (2003), and 
Rayner (1998). 



Saccades and Fixations 

The monocular visual field in a human observer 
extends over 150° in both the horizontal and verti- 
cal directions. The perception of visual detail, how- 
ever, is restricted to a small rod-free region of 
central vision known as the fovea, and in particular, 
to the rod- and capillary-free joveola, a single de- 
gree of visual angle in diameter (Wandell, 1995). 
Away from the fovea, perception of high spatial fre- 
quencies declines rapidly, due in part to increases 



in the spacing between the retinal cones and in 
part to changes in the neural connectivity between 
retina, geniculate, and cortex (Wilson, Levi, Maffei, 
Rovamo, & DeValois, 1990). Movements of the 
eyes are used to shift the small region of detailed 
vision from one area of interest to another as neces- 
sary for visual exploration and monitoring. 

Researchers have identified several distinct 
forms of eye movements (see Findlay & Gilchrist, 
2003, for a detailed review). Three that directly 
subserve visual information sampling are vergence 
shifts, pursuit movements, and saccades. Vergence 
shifts are movements in which the left and right 
eyes rotate in opposite directions, moving the 
point of regard in depth relative to the observer. 
Pursuit movements are those in which the eyes 
travel smoothly and in conjunction so as to main- 
tain fixation on a moving object. Of most interest 
to human factors specialists, finally, are saccades, 
ballistic movements that rapidly shift the observers 
gaze from one point of interest to another. Saccadic 
eye movements are typically 30 to 50 ms in dura- 
tion, and can reach velocities of 500° per second 
(Rayner, 1998). They tend to occur 3-4 times per 
second in normal scene viewing and are separated 
by fixations that are generally 200-300 ms in dura- 
tion. Saccades can be broadly classified as reflexive, 
voluntary, or memory guided. Reflexive saccades, 
though they can be suppressed or modified by in- 
tentional processes, are visually guided movements 
programmed automatically in response to a tran- 
sient signal at the saccade target location. Volun- 
tary saccades are programmed endogenously to a 
location not marked by a visual transient. Memory- 
guided saccades are made to a cued location, but 
only after delay (Pierrot-Deseilligny Milea, & Muri, 
2004). 

Notably, useful visual information is acquired 
only during fixations. During the execution of a sac- 
cade, thresholds for visual detection are highly ele- 
vated and little new information is acquired, a 
phenomenon known as saccadic suppression (Matin, 
1974). This effect seems to occur in part because 
pre- and postsaccadic input masks the signals pro- 
duced during the movement itself (Campbell & 
Wurtz, 1979), in part because the retinal stimulus 
patterns that obtain while the eye is in flight fall 
largely outside the visual systems spatiotemporal 
sensitivity range (Castet & Masson, 2000), and in 
part because retinal signals generated during the 
saccade are inhibited at a site between the retina and 
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the first visual cortical area (Burr, Morrone, & Ross, 
1994; Riggs, Merton, & Morton, 1974; Thilo, San- 
toro, Walsh, & Blakemore, 2004). As such, visual 
sampling during saccadic scanning is essentially dis- 
crete, with periods of information acquisition inter- 
rupted by transient periods of (near) blindness. 
Interestingly, Gilchrist, Brown, and Findlay (1997) 
have described a patient who is unable to move 
her eyes yet shows a pattern of visual scanning 
behavior — discrete dwells interspersed with rapid 
head movements — similar to that seen in normal 
subjects. Gilchrist et al. take this as evidence that 
saccadic behavior may be an optimal method of 
sampling information from a visual scene. 



Describing Saccadic Behavior 

Given the complexity of our visual behavior, it is 
not surprising that a myriad of dependent measures 
for understanding oculomotor scanning have been 
used (for more comprehensive discussion of such 
measures, see Inhoff & Radach, 1998; Jacob & 
Karn, 2003; Rayner, 1998). Typically, saccades are 
described by their amplitude and direction, though 
in some cases saccade duration and maximum ve- 
locity may also be of interest. Analysis of visual 
samples that occur between saccades is more com- 
plex. A sample is characterized in part by its loca- 
tion, obviously, and for some purposes it may be 
necessary to specify location with great precision. 
In human factors research, though, it is often suffi- 
cient merely to determine whether the point of re- 
gard falls within a designated area of interest (AOI). 
For example, an aviation psychologist may wish to 
know which cockpit instrument the pilot is inspect- 
ing at a given moment, without needing to know 
the precise location of the pilot's gaze within the in- 
strument. Thus, multiple consecutive fixations 
within the same AOI might be considered part of a 
single look at the instrument. Any series of one or 
more consecutive fixations on the same AOI is 
termed a dwell or gaze. A sequence of multiple con- 
secutive fixations within a single AOI may occur ei- 
ther because the observer reorients the eyes within 
the AOI (if the region is sufficiently large) or be- 
cause the observer executes a corrective microsac- 
cade (Rayner, 1998) to return the point of regard to 
its original location following an unintended drift. 
It is also common for an observer scanning a scene 
or display to return more than once to the same 



AOI. In such cases, the initial gaze within the AOI 
may be termed the first-pass gaze- 
Given these considerations, there are multiple 
measures for describing the temporal and spa- 
tiotemporal properties of oculomotor sampling. At 
the finest level is fixation duration, the duration of a 
single pause between saccades. As noted above, 
fixation durations are sensitive to manipulations 
that affect the difficulty of extracting foveal visual in- 
formation, including noise masking (van Diepen & 
d'Ydewalle, 2003), contrast reduction (van Diepen, 
2001), and, in a visual search task, a decrease in 
target-distractor discriminability (Hooge & Erke- 
lens, 1996). In cases where there are multiple con- 
secutive fixations on the same AOI, fixation 
duration may be less informative than gaze dura- 
tion. In understanding a pilot's cockpit scanning 
behavior, for instance, it is probably more useful to 
know the total amount of time spent on a gaze at a 
given instrument than to know the durations of in- 
dividual fixations comprising the gaze. It also com- 
mon, finally, to report dwell frequencies for various 
AOIs. Dwell frequency data are of particular inter- 
est in supervisory monitoring tasks, where the op- 
erator will scan a limited set of AOIs over an 
extended period of time. A number of studies have 
shown that gaze duration is sensitive to the infor- 
mation value of the item being fixated, with items 
that are unexpected or semantically inconsistent 
with their context receiving longer gazes than 
items that are expected within the context (e.g., 
Friedman, 1979; Loftus & Mackworth, 1978). In 
the context of aviation, as noted above, gaze dura- 
tions in the cockpit are modulated by the difficulty 
of extracting information from the fixated instru- 
ment, with instruments that are more difficult 
to read or interpret receiving longer dwells (Fitts 
et al., 1950). Dwell frequency, in contrast, is deter- 
mined by the relative importance of each AOI, 
such that a cockpit instrument is visited more often 
when it provides information critical to the flight 
maneuver being performed (Bellenkes, Wickens, & 
Kramer, 1997; Fitts et al., 1950). 



Overt and Covert Attention Shifts 

Interpretation of eye movement data is typically 
guided by the assumption that an observer is 
attending where the eyes are pointing, a premise 
formalized by Just and Carpenter (1980) as the 
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eye-mind assumption. It is also well-known, how- 
ever, that attention can at times be allocated to one 
location within the visual field while the eyes 
are focused on another. Indeed, it is common for 
cognitive psychologists to study attention using 
tachistoscopic displays expressly designed to dis- 
allow eye movements so as to avoid confounding 
the effects of visual acuity with those of central at- 
tentional processes. Attention researchers thus dis- 
tinguish overt shifts of attention, those involving a 
movement of the head, eyes, or body, from covert 
shifts, accomplished without reorienting the sen- 
sory surface (Posner, 1980). 

Of interest is the interaction between overt and 
covert orienting. Posner (1980) described four 
forms that the physiological and functional rela- 
tionship between overt and covert processes might 
take. At one extreme was the logical possibility that 
eye movements and covert attention shifts might 
be produced by the identical neural mechanisms. 
The result would be full functional dependence be- 
tween the two forms of orienting, a prediction dis- 
proved by the observation that attention can be 
oriented covertly without triggering an accompa- 
nying eye movement. At the opposite end of the 
theoretical spectrum was the possibility that eye 
movements and covert shifts are functionally inde- 
pendent, a hypothesis at odds with the eye-mind 
assumption and with the introspective sense that 
attention and the eyes are closely linked. In be- 
tween the possibilities of full dependence and full 
independence, Posner identified the theories of an 
efference-driven relationship and of a nonobliga- 
tory functional relationship. The latter account 
posits that overt and covert attention are not struc- 
turally coupled but tend to travel in concert simply 
because they are attracted by similar objects and 
events. The efference theory, also known as the 
premotor (Rizzolatti, Riggio, Dascola, & Umilta, 
1987) or oculomotor readiness theory (Klein, 1980), 
holds that a covert attention shift is equivalent to 
an unexecuted overt shift. The neural mechanisms 
and processes involved in covert and overt orient- 
ing, in other words, are identical up until the point 
at which an overt shift is carried out. 

Which of these accounts is correct? Although 
early findings appeared to support the theory of a 
nonstructural functional relationship between 
overt and covert orienting (Klein, 1980), more re- 
cent findings have demonstrated that covert atten- 
tion is obligatorily shifted to the location of a 



saccade target prior to execution of the overt move- 
ment, consistent with the claims of the premotor 
theory. For example, Hoffman and Subramaniam 
(1995) asked subjects to saccade to one of four tar- 
get locations (above, below, right, or left of fixa- 
tion) while attempting to discriminate a visual 
probe letter presented prior to saccade onset pre- 
sented at one of the same four locations. Probe dis- 
crimination was best when the probe appeared at 
the saccade target location. Likewise, Deubel and 
Schneider (1996) asked subjects to discriminate a 
target character embedded in a row of three items 
while simultaneously preparing a saccade to one 
of the three locations. Data again showed that 
psychophysical performance was highest when 
the discrimination target and saccade target were 
coincident (see also Sheperd, Findlay, & Hockey, 
1986). Kowler, Anderson, Dosher, and Blaser (1995) 
had subjects report the identity of a letter that ap- 
peared at one of eight locations arranged on an 
imaginary circle while also programming a saccade. 
Performance on both tasks was best when the loca- 
tions for the saccade and letter identification were 
the same. Similar effects have been found across 
stimulus modalities, interestingly, with discrimina- 
tion accuracy enhanced for auditory probes that 
are presented near rather than far from a saccade 
target location (Rorden & Driver, 1999). Evidence 
for a presaccadic shift of covert attention has also 
come from findings of a preview benefit, a tendency 
for information that is available at the saccade 
target location prior to movement execution to fa- 
cilitate postsaccadic processing (Henderson, Pollat- 
sek, & Rayner, 1987). As discussed below, finally, 
positron emission tomography (PET) and func- 
tional magnetic resonance imaging (fMRI) data 
support the conclusion that overt and covert atten- 
tional control are rooted in overlapping neural sys- 
tems. It should be noted that covert attentional 
processes apparently do not serve the purpose of 
scouting out or selecting potential saccade targets 
(Findlay, 1997; Peterson, Kramer, Wang, Irwin, & 
McCarley 2001). Rather, the processes responsible 
for selecting saccade targets appear to operate in 
parallel across the visual field (as captured in the 
models of saccade generation discussed below), 
with covert attention accruing at the target location 
in the lead-up to saccade execution. 

In total, data indicate that a reallocation of 
covert attention is a necessary but not sufficient 
precondition for saccade execution; attention can 
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move without the eyes following, but the eyes can- 
not move unless attention has shifted first. Thus, a 
fixation at a given location is strong evidence that 
attention has been there, but the failure to fixate a 
location does not guarantee that the location has 
not been attended. For the human factors special- 
ist, however, these conclusions come with two 
caveats. First, it is not clear that covert attentional 
shifts independent of eye movements play a signifi- 
cant role in many real-world tasks (Findlay & 
Gilchrist, 1998; Moray & Rotenberg, 1989). In- 
deed, subjects often rely on oculomotor scanning 
even in cases where a stronger reliance on covert 
processing might improve performance. Brown, 
Huey, and Findlay (1997), for example, found a 
tendency for subjects to commence oculomotor 
scanning rapidly following stimulus onset in a vi- 
sual search task, despite the fact that they could 
improve performance by delaying the initial sac- 
cade and allowing greater time for covert processes 
to operate. Similarly, Shapiro and Raymond (1989) 
discovered that although action video game players 
could be trained to improve their scores by execut- 
ing fewer saccades and relying instead on covert 
processing, they did not tend to adopt this strategy 
spontaneously. It may thus be that in many natura- 
listic tasks, covert attentional shifts are by them- 
selves of little practical interest. Moray and 
Rotenberg (1989, p. 1320) speculated that "the 
more a task is like a real industrial task, where the 
operators can move their head, eyes and bodies, 
can slouch or sit up, and can, in general, relate 
physically to the task in any way they please, the 
more 'attention' is limited by 'coarse' mechanisms 
such as eye and hand movements." The second 
caveat is that even if an operator has fixated on an 
item of interest, and therefore has attended to it, 
there is no guarantee that the item has been pro- 
cessed to the point of recognition or access to 
working memory. For instance, "looked but failed 
to see" errors, in which a driver gazes in the di- 
rection of a road hazard but does not notice it, are 
a relatively common cause of traffic accidents 
(Langham, Hole, Edwards, & O'Neill, 2002). Such 
lapses in visual encoding can be engendered by 
nonvisual cognitive load, like that imposed by an 
auditory or verbal loading task. In one demonstra- 
tion of this, Strayer, Drews, and Johnston (2003) 
measured subjects' incidental memory for road 
signs encountered in a simulated driving task. Data 
showed that recognition was degraded if stimulus 



encoding occurred while subjects were engaged in 
a hands-free cell phone conversation. This effect ob- 
tained even when controlling for the amount time 
that the road signs were fixated at the time of en- 
coding, confirming that the encoding of informa- 
tion within a gaze was impaired independent of 
possible changes in gaze duration. 



Attentional Breadth 

In addition to asking questions about movements 
of covert attention within the course of a visual fixa- 
tion, researchers have also measured the breadth 
of covert processing. The terms stationary field 
(Sanders, 1963) , functional field oj view (FFOV), con- 
spicuity area (Engel, 1971), visual lobe (Courtney & 
Chan, 1986), visual span (Jacobs, 1986), and percep- 
tual span (McConkie & Rayner, 1975, 1976) have 
all been used to describe the area surrounding the 
point of regard from which information is extracted 
in the course of a fixation. (Though researchers 
sometimes draw distinctions between their precise 
definitions, we use these various terms interchange- 
ably.) In some cases, the perceptual span is delin- 
eated psychophysically by measuring the distance 
from fixation at which a target can be detected 
or localized within a briefly flashed display (e.g., 
Sekuler & Ball, 1986; Scialfa, Kline, & Lyman, 
1987). In other instances, span is assessed using 
gaze-contingent paradigms in which eye movements 
are tracked and visual displays are modified as 
a function of where the observer is looking 
(McConkie & Rayner, 1975, 1976; Rayner, 1998). 
The most common of these is the moving window 
technique. Here, task-critical information is masked 
or degraded across the display except within a small 
area surrounding the point of regard. An example of 
such a display is shown in figure 7.1. To measure 
the perceptual span, researchers can manipulate the 
size of the moving window and determine the point 
at which performance reaches normal levels. In 
reading, the perceptual span is asymmetrical, with 
the direction of the asymmetry depending on the 
direction in which readers scan. In English, which 
is printed left to right, the perceptual span extends 
3-4 characters to the left of the reader's fixation and 
14-15 characters to the right (McConkie & Rayner, 
1975, 1976). In Hebrew, printed right to left, the 
direction of asymmetry is reversed (Pollatsek, 
Bolozky, Well, & Rayner, 1981). In both cases, in 
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Figure 7.1 . Illustration of a moving window display. Display resolution is normal within a re- 
gion centered on the observer's point of regard and is degraded outside that region. Display 
is gaze-contingent, such that the high-resolution window follows the point of regard as the 
eyes move. From Reingold et al. (2003). From Reingold, E. M., Loschky, L. C, McConkie, 
G. W, & Stampe, D. M. (2003). Gaze-contingent multresolutional displays: An integrative 
review. Human Factors, 45, 307-328, with permission of Blackwell Publishing. 



other words, the perceptual span extends farther in 
the direction that the reader is scanning than in the 
opposite direction. In visual search, the span of ef- 
fective vision depends on the similarity of the target 
and distractor items, being narrower when target 
and distractors are more similar (Rayner & Fisher, 
1987a). Span size can also be reduced by increases 
in nonvisual workload (Pomplun, Reingold, & 
Shen, 2001). Jacobs (1986) found that the size of 
the visual span in a search task, as manipulated 
through changes in target-distractor discriminabil- 
ity, accounts for a large proportion of variance in 
saccade amplitudes but a relatively small propor- 
tion of variance in fixation durations. 

Distinct functional regions can also be delin- 
eated outside the central perceptual span. An ex- 
periment by Sanders (1963) required subjects to 
make same-different judgments of paired stimuli, 
one presented at the subject's initial fixation point, 
the other at a distance of between 10° and 100° of 
visual angle to the right. Data suggested that the vi- 
sual field could be divided into three concentric 
zones. The stationary field, as noted above, was the 
region surrounding fixation within which both 



stimuli could be processed without an eye move- 
ment. The eye field was the region within which an 
eye movement was required from the left item to 
the right one in order to compare them. The head 
field, finally, was the region within which a com- 
bined eye and head movement was necessary to 
compare the two targets. Data showed that moving 
the right stimulus outward from the stationary field 
to the eye field increased response times for same- 
different judgments, and that moving it again from 
the eye field to the head field increased them fur- 
ther still. Importantly, the response times calcu- 
lated by Sanders did not include the time necessary 
for execution of the eye and head movements 
themselves. The transition from stationary field to 
eye field, and then from eye field to head field, 
therefore seems not only to require additional mo- 
tor responses of the observer, but also to impose a 
cost of integrating visual information across multi- 
ple gazes (cf. Irwin, 1996). The processing ad- 
vantage of the eye field relative to the head field 
arose, Sanders speculated, because coarse prelimi- 
nary processing of the information within the eye 
field primed recognition of stimuli that were subse- 
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quently brought within the stationary field (see 
also Rayner & Fisher, 1987b). 

Attentional breadth appears to be an important 
mediator of visual performance in complex tasks 
and real- world domains. Pringle, Irwin, Kramer, 
and Atchley (2001) found a negative correlation 
between the size of the FFOV, as measured with a 
visual search task using a briefly flashed display, 
and the time necessary for subjects to notice visual 
events within free-viewed traffic scenes in a change 
detection task. Their results suggest that a psy- 
chophysical measure of visual span can indeed 
provide an index of how "far and wide" observers 
tend to spread covert attention while scanning a 
complex naturalistic stimulus. Consistent with this 
conclusion, Owsley and colleagues (1998) have 
found negative correlations between the size of 
FFOV and accident rates in elderly drivers. 



Bottom- Up and Top-Down Control 

In developing most any form of visual information 
display — aircraft cockpit, automobile dashboard, 
computer software interface, traffic sign, or warn- 
ing label — one of the designer's primary goals is to 
predict and control the display user's scanning, en- 
suring that task-relevant stimuli are noticed, fix- 
ated, and processed. For this, it is necessary to 
understand the information that guides the selec- 
tion of saccade targets in oculomotor scanning. 
Like the control of covert attention (Yantis, 1998), 
guidance of the eyes is accomplished by the inter- 



play of bottom-up/stimulus-driven and top-down/ 
goal-driven mechanisms. Attention shifts, overt or 
covert, are considered bottom-up to the extent that 
they are driven by stimulus properties indepen- 
dently of the observer's behavioral goals or expec- 
tations, and are deemed top-down to the extent 
that they are modulated by intentions or expecta- 
tions. Among scientists studying covert attention, 
theorists variously argue that attentional shifts are 
never purely stimulus driven but are always regu- 
lated by top-down control settings (Folk, Reming- 
ton, & Johnston, 1992); that attentional control is 
primarily stimulus driven, such that salient objects 
tend to capture covert processing regardless of the 
observer's goals or set (Theeuwes, 1994); and that 
some stimulus properties — specifically, the abrupt 
appearance of a new object within the visual 
scene — are uniquely capable of capturing covert 
attention, but that salience per se is not sufficient 
to trigger a reflexive attention shift against the ob- 
server's intentions (Jonides & Yantis, 1988; Yantis 
& Jonides, 1990). Despite much research, debate 
among the proponents of these competing theories 
has not been resolved. 

In oculomotor research, bottom-up control has 
been studied using the oculomotor capture paradigm 
introduced by Theeuwes, Kramer, Hahn, and Irwin 
(1998). Stimuli and procedure from a typical ex- 
periment are illustrated in figure 7.2. Subjects in 
the study by Theeuwes et al. began each trial by 
gazing at a central fixation marker surrounded by a 
ring of six gray circles, each containing a small gray 
figure-8 premask. After a brief delay, five of the cir- 
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Figure 7.2. Schematic illustration of the stimuli employed in the Theeuwes et al. (1998) study of 
oculomotor capture. Gray circles are indicated by dashed lines; red circles are indicated by solid lines. 
See text for description of procedure. Reprinted with permission of Blackwell Publishing from Theeuwes 
et al. (1998). 
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cles changed to red and segments were removed 
from premasks to reveal letters. Simultaneously, an 
additional red circle containing a letter appeared 
abruptly at a location that had previously been un- 
occupied. The subjects' task was to saccade as 
quickly as possible to the remaining gray circle and 
identify the letter within it. The onset circle was 
never itself the target. Despite this, remarkably, on 
roughly a quarter of all trials subjects made a re- 
flexive saccade to the onset circle before moving 
to the gray target stimulus. Just as the sudden ap- 
pearance of a new visual object seems to strongly 
and perhaps obligatorily attract covert attention 
(Yantis Srjonides, 1990), the appearance of a new 
distractor object captured the subjects' gaze, over- 
riding top-down settings that specified the gray 
item as the desired saccade target. Figure 7.3 pres- 
ents sample data. Control experiments confirmed 
that the additional distractor captured attention 
only when it appeared as an abrupt onset, not 
when it was present from the beginning of the trial. 
On those trials where capture occurred, interest- 
ingly, the eyes dwelt on the onset stimulus for less 
than 100 ms before shifting to the gray target, a pe- 
riod too brief to allow full programming of a sac- 
cade. The authors concluded that programming of 
two saccades — one a stimulus-driven movement 
toward the onset object and the other a goal-driven 
movement toward the gray target — had occurred 
in parallel (Theeuwes et al, 1998). Subsequent ex- 
periments have suggested that abrupt onsets are 
uniquely effective in producing oculomotor cap- 
ture. Luminance increases and color singletons (i.e., 
uniquely colored items among homogeneously col- 
ored background stimuli), for example, produce 
far weaker capture of the eyes, even when matched 
in salience to an abrupt onset (Irwin, Colcombe, 
Kramer, & Hahn, 2000). 

Another simple and popular method for study- 
ing bottom-up and top-down control of eye move- 
ments is the antisaccade task developed by Hallett 
(1978). The observer begins a trial by gazing at a 
central fixation mark, and a visual go signal is then 



Figure 7.3. Sample data from Theeuwes et al. (1998). 
Data points indicate gaze position samples collected 
at 250 Hz during the first saccade of each trial follow- 
ing target onset. The target stimulus appears at the 
11 o'clock position in all frames. Reprinted with per- 
mission of Blackwell Publishing from Theeuwes et al. 
(1998). 
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flashed in either the left or right periphery. The 
observers task is to move the eyes in the direction 
opposite the go signal. To perform the task correctly, 
therefore, the observer must suppress the tendency 
to make a reflexive saccade toward the go signal and 
instead program a voluntary saccade in the other di- 
rection. Performance on antisaccade trials can be 
compared to that on prosaccade trials, where the ob- 
server is asked to saccade toward the transient sig- 
nal. Data show that antisaccades are initiated more 
slowly than prosaccades, and that directional errors 
are common in the antisaccade conditions but rare 
in the prosaccade task (Everling & Fischer, 1998; 
Hallett, 1978). As discussed further below, Guitton, 
Buchtel, and Douglas (1985) found a selective 
deficit in antisaccade performance among patients 
with frontal lobe lesions, suggesting a role for the 
frontal regions in inhibiting reflexive saccades and 
programming voluntary movements. Consistent 
with this conclusion, data from healthy subjects in- 
dicate that antisaccade performance is mediated by 
executive working memory processes typically as- 
cribed to the frontal lobes (Kane, Bleckley, Conway, 
& Engle, 2001; Roberts, Hager, & Heron, 1994). 

Search is another form of visual behavior in- 
corporating bottom-up and top-down processes. 
Modern theories of search (Itti & Koch, 2000; 
Treisman & Sato, 1990; Wolfe, 1994) generally as- 
sume that attentional scanning is guided by activa- 
tion within a salience map (Koch & Ullman, 1985) 
representing points of interest within the visual 
scene. In the first stages of search, a scene or dis- 
play is encoded within an array of low-level repre- 
sentations, known as feature maps, tuned to 
stimulus properties such as color, orientation, mo- 
tion, and spatial frequency. Contrast between fea- 
ture values (e.g., contrast between red and green 
regions on either side of a chromatic border) pro- 
duces bottom-up signals that are fed forward to the 
salience map and serve to direct attention within 
the visual scene. In cases where physical properties 
of the target are specified, goal-driven control of 
search can be effected by top-down modulation of 
activation within feature maps, amplifying signals 
within the maps that encode known target proper- 
ties or attenuating activation within the maps that 
encode properties that do not belong to the target 
(Treisman & Sato, 1990; Wolfe, 1994). The influ- 
ence of feature-guided search is evident in the phe- 
nomenon of saccade selectivity during visual search, 
the tendency for observers to preferentially fixate 



stimuli that share features with the target (e.g., Find- 
lay, 1997; Scialfa & Joffe, 1998; D. E. Williams & 
Reingold, 2001; L. G. Williams, 1967). A number 
of studies have found that saccade guidance based 
on color is more effective than that based on shape 
or orientation (D. E. Williams & Reingold, 2001; 
L. G. Williams, 1967), presumably because of the 
poor spatial resolution in the peripheral visual 
field. Under circumstances where target uncer- 
tainty or high levels of camouflage make top-down 
feature guidance ineffective, models based on 
bottom-up guidance alone may predict scanning 
performance well (Itti & Koch, 2000). Another form 
of top-down control in search, independent of 
feature-based guidance, is contextual cuing, whereby 
attention is biased toward potential target-rich loca- 
tions within a familiar scene (Chun & Jiang, 1998; 
Peterson & Kramer, 2001). Interestingly, this effect 
occurs even within abstract stimulus displays (e.g., 
random arrangements of letters) and without con- 
scious recognition that a given scene has been previ- 
ously encountered. The phenomenon thus appears 
to be driven by implicit memory for specific scene 
exemplars, rather than by explicit semantic knowl- 
edge of likely target locations. 

The influence of both goal-driven and stimulus- 
driven processes are evident in naturalistic and 
complex tasks, though to varying degrees across do- 
mains. A famous demonstration of top-down eye 
movement guidance came from Yarbus (1967), who 
showed that observers will scan a picture differently 
depending on the judgment they are asked to 
make of it (e.g., judging whether the family de- 
picted in a scene is wealthy versus judging the fam- 
ily members' ages). In scanning images of traffic 
scenes, observers are more likely to fixate on loca- 
tions or objects that are highly task relevant (e.g., 
traffic lights, pedestrians) than items of little task 
relevance, but at the same time are more likely to 
fixate on items that are highly salient than on items 
that are less conspicuous (McCarley, Vais, et al., 
2004; Pringle, 2001; Theuwes, 1996). In supervisory 
monitoring, operators adapt their visual sampling to 
the bandwidth and information value of various 
channels, apparently forming a mental model of 
system behavior to guide their scanning top-down 
(Carbonell, Ward, & Senders, 1968; Senders, 1964). 
In performing stereotyped everyday tasks in non- 
dynamic environments, similarly, operators may 
rely almost exclusively on goal-driven sampling. 
Land and Hayhoe (2001), for example, recorded 
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eye movements of subjects making sandwiches or 
preparing tea. Data showed a close coupling of eye 
movements to manipulative actions. Subjects car- 
ried out each task by performing a series of object- 
related actions (ORAs), each a simple manual act 
that brought the subject a step closer to the final 
goal. ORAs were typically delineated by a shift of the 
eyes from the object currently being manipulated 
(e.g., slice of bread) to the object next to be manipu- 
lated (e.g., knife). A fixation on each object generally 
began a fraction of a second before manipulation of 
the object was initiated, and persisted until gaze 
shifted to a new object to begin the next ORA. In 
both the sandwich-making and tea-making tasks, 
the proportion of fixations on task-irrelevant objects 
was less than 5%, leading the authors to conclude 
that eye movements were strongly goal driven. 



Effort 

Although visual scanning is guided largely by in- 
formation content and information demand, as re- 
flected in the stimulus-driven and goal-driven 
control of the eyes, it may also be constrained by 
information access cost, the effort needed to reach 
and sample a given channel. Gray and Fu (2004) 
noted that even small differences in the time or ef- 
fort needed to complete a given action can prompt 
changes in an operators behavioral strategy, pro- 
ducing large changes in overall performance levels. 
In complex visuomotor tasks, operators appear to 
trade the costs of sampling movements against the 
cognitive demands of maintaining information in 
working memory. A series of experiments by Bal- 
lard, Hayhoe, and Pelz (1995) illustrate this well. 
The subjects' task was to arrange colored tiles to 
create a copy of a mosaic pattern provided by the 
experimenters. Displays were divided into three ar- 
eas: the model, the pattern to be copied; the re- 
source, the space from which the colored tiles to be 
placed were retrieved; and the workspace, the re- 
gion in which the copy was to be constructed. To 
perform the task, subjects were required to exam- 
ine and remember the model, and then to retrieve 
colored tiles from the resource and place them 
within the workspace as appropriate. As noted by 
the experimenters, the task thus tapped a variety of 
sensory, cognitive, and motor skills, but was simple 
enough that moment-to-moment subgoals (e.g., 
pick up tile, drop tile) could be identified. Most 



important, the task allowed participants to adopt 
their own performance strategies. The experimenters 
could thus examine how variations in the demands 
of the task altered subjects' strategy selection. 

In a first experiment (Ballard et al., 1995), the 
model, resource pool, and workspace were pre- 
sented on a single computer monitor, and subjects 
performed the task with their heads fixed in a chin- 
rest. All visual information needed to complete 
each trial could be obtained with eye movements 
alone. Under these circumstances, subjects ap- 
peared to trade the costs of frequent eye move- 
ments for the benefits of a low working memory 
load. Indeed, data suggested that subjects held no 
more than the minimum possible amount of infor- 
mation in visual working memory. Rather than at- 
tempting to remember the model as a whole, or to 
simultaneously remember both the color and loca- 
tion of the next block to be placed, subjects tended 
to fixate the model once before selecting each 
block from the resource pool, to gaze at the re- 
source pool in order to pick up a block, then to 
look at the model again before placing the selected 
block. A similar pattern of effects obtained in later 
experiments in which subjects performed the task 
by manipulating 3-D blocks, rather than computer 
icons, as long as the model, workspace, and re- 
source pool were close enough (within 20° of vi- 
sual angle) together for subjects to scan between 
them using eye movements and short head move- 
ments. Results changed, however, when the model 
and workspace were separated by a greater dis- 
tance (70° of visual angle), such that large head 
movements were necessary to scan between them. 
In these cases, subjects relied more heavily on vi- 
sual working memory, gathering and remembering 
a greater amount of information with each look at 
the model and therefore reducing the need for 
long-distance, high-effort gaze shifts between the 
model and the workspace. The implication of such 
findings for designers of human-machine systems 
are direct: To minimize demands on scanning and 
working memory, display channels that provide in- 
formation to be compared or integrated should be 
arrayed near one another, and controls should be 
placed near the displays that they affect. These 
guidelines, long recognized in human factors re- 
search, have been codified as the proximity display 
principle (Wickens & Carswell, 1995) and coloca- 
tion principle (Wickens & Hollands, 2000) of inter- 
face design. 
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Neurophysiology of Saccadic Behavior 

Although the use of eye movements in the study 
of behavior and information processing has en- 
hanced our understanding of human performance 
and cognition in real-world tasks, this research has 
to a large extent progressed in parallel with the 
study of the neuronal underpinnings of eye move- 
ments and their relationship to different cognitive 
constructs. Given the view of neuroergonomics 
(Parasuraman, 2003) that each domain of research 
may benefit from an understanding of the other, it 
is useful to review research that has addressed the 
relationship between eye movements and neu- 
ronal activity (see reviews by Gaymard, Ploner, Ri- 
vaud, Vermersch, & Pierrot-Deseilligny, 1998; 
Pierrot-Deseilligny et al. , 2004, for additional 
details). 

Researchers have utilized a variety of tech- 
niques to study the neural control of saccades in 
humans. Early research on humans attempted to lo- 
calize specific saccade control functions by examin- 
ing patients with circumscribed lesions. The research 
of Guitton et al. (1985), mentioned above, revealed 
deficits in the antisaccade but not the prosaccade 
task among patients with frontal lesions. These data 
and others from patient studies have been inter- 
preted as evidence for the role of the frontal regions, 
and more specifically the dorsolateral prefrontal 
cortex (DLPFC) and frontal eye fields, in the inhibi- 
tion of reflexive saccades and the programming and 
execution of voluntary saccades. 

More recent studies have employed neu- 
roimaging techniques such as PET and fMRI to ex- 
amine the neuronal circuits that generate saccades, 
exploring the network of cortical and subcortical 
regions that contribute in common and uniquely to 
the planning and execution of prosaccades, anti- 
saccades, memory-based saccades, and covert at- 
tention shifts. These studies have, by and large, 
found evidence for the activation of extensive net- 
works of frontal, parietal, and midbrain regions, 
similar to areas revealed in studies of single-unit ac- 
tivity with nonhuman primates (Gottlieb, Kusunoki, 
& Goldberg, 1998) in different eye movement and 
attention-shifting tasks. Corbetta (1998) was the 
first to show, using fMRI, that the brain regions as- 
sociated with covert orienting and overt eye move- 
ments overlap in frontal and parietal cortices. 
Kimmig et al. (2001) found, in a PET study, that 
similar brain regions were activated in pro- and 



antisaccade tasks but that activation levels were 
higher in the antisaccade task for areas including 
the frontal eye fields, supplementary eye fields, pari- 
etal eye fields, putamen, and thalamus. Sweeney 
et al. (1996) reported increased activation in the 
DLPFC, a brain region associated with working 
memory and interference control, in both a 
memory-guided saccade and antisaccade task. 

Another technique useful in the study of the 
neuronal underpinnings of saccades is transcranial 
magnetic stimulation (TMS). TMS involves the ap- 
plication of a brief magnetic pulse to the scalp. 
This pulse induces localized electrical fields that al- 
ter the electrical field in the brain below the stimu- 
lator, in effect producing a virtual lesion that is 
both reversible and transient. Terao et al. (1998) 
administered a TMS pulse, over either the frontal 
or parietal cortex, at various times after the presen- 
tation of a peripheral stimulus that cued an antisac- 
cade. Increased saccadic latencies and erroneous 
prosaccades (i.e., saccades toward rather than away 
from the eliciting stimulus) were induced by stim- 
ulation of either cortical region, though changes in 
saccade parameters occurred earlier for parietal 
than for frontal stimulation. Ro, Henik, Machado, 
and Rafal (1997) employed TMS to examine the 
neuronal organization of stimulus-driven and vol- 
untary saccades. Subjects were asked to make sac- 
cades to either the left or right in response to either 
a centrally located arrow (endogenous go signal) or 
a peripheral (exogenous go signal) marker. TMS 
pulses were presented at varying intervals with re- 
spect to the saccade go signal. TMS delivered over 
the superior prefrontal cortex increased latencies 
for saccades made in response to the endogenous 
go signal. No effects were observed for saccade la- 
tency when the go signal was exogenous (see also 
Muri, Henik, Machado, & Rafal, 1996). These data 
suggest that TMS can be employed to study both 
the locus of different saccade control and imple- 
mentation functions and the timing of these con- 
trol processes. 

The studies described above, along with a 
much more extensive body of research that has ex- 
amined eye movements in humans as well as other 
animals, have begun to map out the neural circuits 
that are responsible for oculomotor control. Results 
indicate that a large number of interconnected 
frontal, parietal, and midbrain regions contribute to 
oculomotor behavior, with different subsets of these 
areas being responsible for the performance of 
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prosaccade, antisaccade, and memory-guided sac- 
cade tasks. For example, visually guided saccades 
(e.g., prosaccades) involve a pathway from the vi- 
sual cortex through the parietal cortex, to the 
frontal and supplementary eye fields, to the supe- 
rior colliculus, and finally to the brain stem, where 
motor commands are generated for eye movements 
(Pierrot-Deseilligny et al., 2004). More direct path- 
ways are also available from the parietal regions and 
separately from the frontal regions to the superior 
colliculus and from the frontal eye fields directly to 
the brain stem (Hanes & Wurtz, 2001). Although 
little is presently known about the functional differ- 
ences among these pathways, it does appear that 
the direct pathway from the parietal regions is con- 
cerned with eye movements to salient stimuli of rel- 
evance to the organism and that the direct pathway 
from DLPFC to the superior colliculus may play an 
important role in the inhibition of reflexive sac- 
cades that compete with voluntary saccades. 

An interesting question is whether this knowl- 
edge of the neural circuits that underlie different 
varieties of saccades (i.e., prosaccades, antisac- 
cades, memory-based saccades) will be of use in 
modeling and predicting multitask performance 
decrements in situations of interest to human fac- 
tors researchers and practitioners. A specific char- 
acteristic of neuroergonomics research as described 
by Parasuraman (2003) is the use of knowledge of 
brain mechanisms underlying perception and cog- 
nition to formulate hypotheses or inform models in 
human factors research. For example, models of 
multitask performance (e.g., Just, Carpenter, & 
Miyake, 2003; Kinsbourne & Hicks, 1978; Poison 
& Freidman, 1988; Wickens, 1992) have success- 
fully utilized knowledge of brain function and 
structure in predicting processing bottlenecks in 
complex tasks. Whether the level of specificity of 
neural circuits controlling eye movements will en- 
able the further refinement of multitask models as 
well as other psychological constructs relevant to 
human factors (e.g., visual search, linguistic pro- 
cessing, decision making, etc.) is an interesting and 
important topic for future research. 



Computational Models 
of Saccadic Behavior 

A partial understanding of the pathways that con- 
tribute to oculomotor behavior has led to the re- 



cent development of a number of models of eye 
movement control that attempt to account for a va- 
riety of well-established phenomena. These models 
can be distinguished on at least three dimensions: 
whether they are local versus global models of the 
oculomotor system (i.e., the extent to which the 
model focuses on specific components of the ocu- 
lomotor circuit versus the entire system of neural 
components); the extent to which they are con- 
strained by current knowledge about the physiol- 
ogy and anatomy of oculomotor circuits; and 
whether they are qualitative or quantitative in na- 
ture. For example, Trappenberg, Dorris, Munoz, 
and Klein (2001) described a model of the interac- 
tion of multiple signals in the intermediate levels of 
the superior colliculus leading to the initiation of 
saccades. The model is built on the integration and 
competition of exogenous and endogenous inputs 
via short-distance excitatory and long-distance in- 
hibitory connections between the receptive fields 
of neurons in the superior colliculus. Exogenous 
signals refer to minimally processed visual inputs, 
such as motion and luminance transients, while 
endogenous signals refer to inputs based on in- 
structions and expectancies. The importance of 
both of the model's components, input type and 
nature of connections in the colliculus, are well es- 
tablished in the literature. The model has been 
used to successfully account for both neuron firing 
rates in the superior colliculus of nonhuman ani- 
mals and human performance data in a variety of 
different tasks and phenomena, including the gap 
effect (i.e., faster saccadic reaction times when a 
fixation point is turned off a couple hundred mil- 
liseconds before a prosaccade target is presented as 
compared to a situation in which the fixation stim- 
ulus is maintained), distractor effects on saccadic 
reaction times, the influence of target expectancies 
on saccadic reaction times, and the difference in la- 
tencies between pro- and antisaccades. A model by 
Godjin and Theeuwes (2002) extended the Trap- 
penberg model by adding an additional top-down 
inhibitory mechanism that can diminish the activa- 
tion level of some stimuli and locations based on 
experience and expectancies concerning task- 
relevant stimuli. This model has been successful in 
accounting for the saccade behavior associated 
with the oculomotor capture paradigm. 

Itti and Koch (2000) designed a computational 
model in which shifts of attention (i.e., eye move- 
ments) are based solely on a bottom-up salience- 
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based analysis of the visual world. Local differences 
in orientation, intensity, and color are combined 
into a master salience map that determines the 
order in which locations are inspected. Although 
efforts to incorporate goal-driven search will likely 
improve the model's applicability and perfor- 
mance (Navalpakkam & Itti, 2005), the model in 
its original form is interesting and unique in that 
top-down information such as expectancies and 
experience plays no role in the guidance of the 
eyes. Another relatively unique aspect of this effort 
is that the authors have tested their model with 
both simple laboratory-based visual displays and 
high-resolution photographs of natural scenes. Re- 
sults have been promising. 

A recently proposed model of oculomotor be- 
havior takes a different tack in modeling saccade 
control by describing the role of both cortical and 
subcortical control circuits for a variety of different 
phenomena. Findlay and Walker (1999) proposed 
a model, like that of Trappenberg et al. (2001), that 
entails competitive interaction among different sig- 
nals in the initiation and control of eye move- 
ments. However, in their model the competition 
and integration takes place between inputs con- 
cerned with both the where and the when of eye 
movement initiation throughout the eye movement 
circuit. Predictions of the model are discussed 
in terms of phenomena similar to those examined 
by Trappenberg and colleagues. The Findlay and 
Walker model is notable in its attempt to describe 
how different cortical and subcortical components 
of the oculomotor circuit interact to produce sac- 
cades. Unlike the Trappenberg et al. model, Findlay 
and Walker's model does not yet provide quantita- 
tive predictions that can be used to predict patterns 
of neuronal activity. 

In summary, although there are clearly many 
unanswered questions concerning the neuronal cir- 
cuits that underlie oculomotor control under differ- 
ent circumstances as well as the relation between 
oculomotor control and other forms of exploration 
(e.g., limb movements), we do currently have a suf- 
ficiently detailed understanding of the important 
neuronal circuits and functions to enable us to 
model and predict oculomotor behavior. Indeed, in 
principle there is no reason why the models de- 
scribed above should not be applied to oculomotor 
behavior in more complex tasks such as looking for 
defects in manufactured products or scanning an 
image for a camouflaged military vehicle (see Itti & 



Koch, 2000, for an initial attempt to do this). Appli- 
cations to complex tasks such as these should pro- 
vide useful information on the scalability of these 
models to situations outside the laboratory. 



Conclusions and Future Directions 

Our understanding of the role of eye movements 
in information extraction from displays and real- 
world scenes has increased substantially in recent 
years. Techniques such as gaze-contingent control 
procedures have been refined so as to enable re- 
searchers to infer operators' strategies and capabili- 
ties as they inspect the visual world, either in the 
service of an intended action or for the purpose of 
extracting information that will be encoded into 
memory for later use. 

The near future is likely to bring theoretical 
and technical developments that will further en- 
hance the utility of eye movement technology for 
the understanding of human perception, cognition, 
and action in complex simulated and real-world 
environments. There is an increasing trend for the 
development of multimodal assessment techniques 
(e.g., event-related potentials[ERPs], optical imag- 
ing, heart rate, respiration, etc.) that will capitalize 
on the relative advantages of different measures of 
cognition — in and out of the laboratory. Methods 
such as fMRI, electroencephalogram, ERPs, near- 
infrared spectroscopy, ultrasonography, and others 
in different chapters of this volume will continue to 
be useful. High-speed measurement of eye move- 
ments provides a significant addition to the tool- 
box of neuroergonomics. Given the usefulness of 
eye movement measurement techniques in infer- 
ring cognitive state, the collection and analysis of 
oculomotor data are likely to be an important com- 
ponent in such multimodal effects. The computa- 
tional models described above are also likely to be 
further tested and developed such that they can be 
used in a predictive fashion to enhance the design 
of display devices for systems such as aircraft, auto- 
mobile, and industrial systems. Finally, given the 
increasing need for real-time assessment of percep- 
tual and cognitive state along with the capabilities 
of eye movement measurement procedures to tap 
system-relevant cognitive processes, we anticipate 
that the measurement of oculomotor parameters 
will be integrated into real-time workload and per- 
formance assessment algorithms. 
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Main Points 

1 . Eye movements pervade visual behavior. They 
are important to understand as an element of 
human performance in their own right, and 
also as a window on the perceptual and 
cognitive processes underlying behavioral 
performance. 

2. A visual scene is typically inspected with a 
series of discrete fixations separated by rapid 
saccadic eye movements. Information is 
collected only during the fixations; visual 
input is suppressed during the movements 
themselves. 

3. It is possible to shift visual attention without 
making an eye movement. In everyday visual 
behavior, however, attention and the eyes are 
closely coupled. Performance may be limited 
by the breadth with which the operator 
spreads attention during a fixation. 

4. Eye movements are guided by both bottom- 
up/stimulus-driven and top-down/goal-driven 
processes. Oculomotor behavior may also be 
influenced by information access costs, the 
effort required to scan and sample information 
from the environment. 

5. Eye movements are controlled by a broad 
network of cortical and subcortical brain 
regions, including areas of the parietal cortex, 
the prefrontal cortex, and the superior 
colliculus. 

6. Computational models of eye movement 
control may be useful for guiding the design of 
visual displays and predicting the efficacy of 
visual scanning in applied contexts. 
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The Brain in the Wild 

Tracking Human Behavior in Natural 
and Naturalistic Settings 



A problem for students of human behavior is that 
people often act differently in controlled labora- 
tory and clinical settings than they do in real life. 
This is because the goals, rewards, dangers, bene- 
fits, and time frames of sampled behavior can 
differ markedly between the laboratory and clinic 
and "the wild." A laboratory test may seem ar- 
tificial and frustrating, and may not be taken 
seriously, resulting in a misleadingly poor perfor- 
mance. On the other hand, subjects may be on 
their best behavior and perform optimally when 
they know they are being graded in a laboratory 
or clinic, yet they may behave ineffectively in real 
life and fail to meet their apparent performance 
potentials at work, home, school, or in a host of 
instrumental activities of daily living, such as au- 
tomobile driving. Solutions to these pitfalls in the 
study of brain-behavior relationships, as we shall 
see, can be derived through rigorous observations 
of people at work and play in naturalistic settings, 
drawing from principles already being applied 
in studies of animal behavior and making use of 
great advances in sensor technology for simultane- 
ously recording the movements of individuals, 
their surroundings, and their internal body and 
brain states. 



Measuring Movement 
in the Real World 

In the absence of field observations, most of what 
we know about human behavior in the wild comes 
from human testimony (from structured and un- 
structured interviews and questionnaires) and 
epidemiology — a partial and sometimes inaccurate 
account of "yesterday's history." 

Questionnaire tools may be painstakingly de- 
veloped to assess all manner of behavioral issues 
and quality of life, and these generally consist of 
self-reports of subjects or reports by their family 
members, friends, supervisors, or caregivers. Inci- 
dent reports of unsafe outcomes at work, in hospi- 
tals, or on roads completed by trained observers 
(e.g., medical professionals, human factors experts, 
the police) are another source of similar informa- 
tion. However, these reports may be inaccurate 
because of poor observational skills or bias by 
untrained or trained observers. Subject reports 
may be affected by misperception, misunderstand- 
ing, deceit, and a variety of memory and cognitive 
impairments, including lack of self- awareness of 
acquired impairments caused by fatigue, drugs, ag- 
ing, neurological or psychiatric disease, or systemic 
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medical disorders. These information sources pro- 
vide few data on human performance and physiol- 
ogy and often lack key details of what real people 
do in the real world. 

Under these circumstances, it is important to 
consider more direct sources of evidence of human 
behavior in the real world. It would be advanta- 
geous to combine sensors that capture the move- 
ment, physiology, and even social interactions of 
people who are seeing, feeling, attending, deciding, 
erring, and self-correcting during the activities of 
daily living. Such a "people tracker" could provide 
detailed information on the behavior, physiology, 
and pathophysiology of individuals in key every- 
day situations, in settings, systems, and organiza- 
tions where things may go wrong. 

A people tracker system might be compared to 
a black box recorder in an aircraft or automobile, 
recording behavior sequences leading to an out- 
come or event of interest, such as an error in a task. 
Subjects will be less likely to behave out of charac- 
ter if the system is unobtrusive. For example, indi- 
viduals driving unobtrusive instrumented vehicles 
(see below) may display personal grooming and 
hygiene habits, such as rhinotillexomania (nose 
picking). In this vein, privacy issues are an impor- 
tant consideration. No one should experience the 
merciless voyeurism and intrusions of privacy in- 
flicted on the hapless character played by Jim Car- 
rey in The Truman Show. Instead, the intention of 
such devices would be to prevent injury and im- 
prove human performance and health by develop- 
ing alerting and warning systems, training 
programs, and rehabilitation interventions. 



The Mismatch between Clinical Tests, 
Self-Report, and Real-Life Behavior 

Performance measures obtained in a laboratory or 
clinic may inaccurately reflect real-world perfor- 
mance. People often act differently in the real 
world than they or their relatives indicate. That is, 
they do not do what they say they do, or would do. 
Consider the challenge of assessing the real- 
world potential of individuals with decision- 
making impairments caused by brain damage, 
drugs, fatigue, or developmental disorders. Deci- 
sion making requires the evaluation of immediate 
and long-term consequences of planned actions, 
and it is often included with impulse control, in- 



sight, judgment, and planning under the rubric of 
executive functions (e.g., Benton, 1991; Damasio, 
1996, 1999; Rolls, 1999, 2000). Impairments of 
these functions affect tactical and strategic deci- 
sions and actions in the real world. Some of these 
impaired individuals have high IQs and perform 
remarkably well on cognitive tests, including spe- 
cific tests of decision making, yet fail repeatedly in 
real life. One reason for this mismatch is that labo- 
ratory tests of cognition (measured by standardized 
neuropsychological tests) are imperfect and may 
not measure what we think they do. Another factor 
is that people perform differently when they are 
being observed directly. Furthermore, some individ- 
uals who appear to demonstrate the ability to gener- 
ate good plans may not enact these plans in the real 
world because they lack discipline, motivation, or 
social support, or choose alternative strategies with 
short-term benefits that are disadvantageous in the 
long term. A well-described example is subject EVR, 
who had bilateral frontal lobe amputations associ- 
ated with surgery to remove a meningioma (a non- 
malignant brain tumor). The brain damage in EVR 
resembles that in the famous case of railroad fore- 
man Phineas Gage, whose frontal lobes were se- 
verely injured in 1848 by a large iron rod that blew 
through his head as he was using it to tamponade 
dynamite (Damasio, Grabowski, Frank, Galaburda, 
& Damasio, 1994), transforming him from a trust- 
worthy, hard working, dependable worker to a capri- 
cious, irresponsible, ne'er-do-well. 

Behavior clearly depends very much on the en- 
vironment in which it is observed. A recent study 
of cardiac rehabilitation measures after a myocar- 
dial infarction (e.g., amount a patient walked in 6 
minutes) did not correlate well with patient activity 
on discharge (Jarvis &Janz, 2005). Some of the in- 
dividuals with the best scores in the hospital were 
among the least active at home and vice versa, rais- 
ing concerns that patients with worse hearts were 
too active too soon, and that those with better 
hearts were returning to bad habits at home. Along 
similar lines, U.S. soldiers addicted to heroin under 
stress of war and cheap, pure drugs in Vietnam 
often abstained without treatment back home, 
among family, work, and costly drugs. 

It is becoming increasingly evident that data 
collection in a naturalistic setting is a unique source 
for obtaining critical human factors data relevant to 
the brain at work in the wild. As we shall see, there 
are various issues of device development, sensor 
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choice, and placement. There are needs for tax- 
onomies for classifying devices and for classifying 
likely behavior from sensor outputs. To infer higher- 
level behaviors from sensor and video data in hu- 
mans in naturalistic settings, it is also possible to 
apply ethological techniques that have been used to 
analyze behavior sequences in animals. These pos- 
sibilities open large new areas of research to benefit 
human productivity, health, and organizational sys- 
tems, and can build on research that has addressed 
animal behavior (Prete, 2004). 



Ethology and Remote Tracking 

The output of people trackers may be assessed us- 
ing principles and methods now used to study ani- 
mal behavior. Automated behavior monitors are 
widely used in laboratory neuroscience research for 
measuring general motor activity, tracking animals 
in a testing chamber (such as a maze), and distin- 
guishing basic patterns of behavior (such as walk- 
ing versus rearing in an open-field environment). 
Video and sensor-based tracking also plays a vital 
role in movement sciences, such as kinesiology and 
motor control, for providing detailed, quantitative 
records of movement of different body and limb 
segments during coordinated action. Indeed, most 
modern neuroscience laboratories involve some 
form of automated recording of behaviorally rele- 
vant data from external devices or physiological 
sensors placed on or implanted within the animal. 
As technologies for monitoring animals in lab- 
oratory environments improve, we can expect that 
applications in real-world environments also will 
expand. Basic methods of ethology — the study 
of animal behavior in natural settings, pioneered 
by the Nobel laureates Konrad Lorenz, Niko Tin- 
bergen, and Karl von Frisch — involve direct ob- 
servation of behavior, including descriptive and 
quantitative methods for coding and recording be- 
havioral events. Paper-and-pencil methods of the 
early ethologists have largely been replaced by lap- 
top computer and PDA-based event-recording sys- 
tems. Several software packages now are available 
that are optimized for coding behavior in real time 
or from video recordings, using focal subject, one- 
zero, instantaneous, or other structured sampling 
techniques (Lehner, 1996). Commercial software 
such as The Observer (Noldus Information Tech- 
nology) offers semiautomated assistance to facili- 



tate coding and analysis of behavior; public do- 
main alternatives such as JWatcher (http://galli 
form.psy.mq.edu.au/jwatcher) also are available. 
Event-recording software can be interfaced with 
video or automated sensor data to provide syn- 
chronized records of behavior and physiology that 
are essential for linking overt actions to underlying 
mechanisms. 

More recently, observational methods for 
describing behavior have been augmented by a va- 
riety of recording and remote monitoring technolo- 
gies, including film and video; telemetry systems 
for monitoring basic physiological parameters such 
as heart rate, temperature, respiratory rate, or mus- 
cle activity; and radio transmitters for tracking large- 
scale movements within a home range or during 
migration. 

The power and limitations of measuring be- 
havior from remote sensor data may be well il- 
lustrated by studies of behavior of animal fetuses, 
which are not amenable to direct observation ex- 
cept through invasive procedures. In large animal 
species, such as sheep, arrays of recording instru- 
ments may be surgically placed within the uterine 
environment to measure key variables of behav- 
ioral relevance, including electroencephalogram 
(EEG) or electrocorticogram (ECoG), eye move- 
ments (electrooculogram, EOG), heart rate, blood 
pressure, intratracheal pressure (to detect fetal 
breathing), and electromyogram (EMG) of select 
muscles to measure swallowing, oral activity, limb 
activity, or changes in tone of postural muscles 
(useful for distinguishing quiet and active sleep 
states). Additional recording instruments provide 
data about key environmental conditions in utero, 
such as uterine contractions, partial pressure of 
oxygen (p0 2 ), maternal heart rate, and blood pres- 
sure. After placement of the instruments during 
a surgical preparation, the fetus and uterus are 
returned to the maternal abdomen, and the out- 
put of the recording instruments is routed to an 
automated data acquisition system, which recti- 
fies, averages, and saves measurements within con- 
venient time units (Robinson, Wong, Robertson, 
Nathanielsz, & Smotherman, 1995; Towell, 
Figueroa, Markowitz, Elias, & Nathanielsz, 1987). 

The result of this automated monitoring ap- 
proach is a continuous, real-time record of behav- 
ioral and physiological data, collected 24 hours a 
day 7 days a week, over the last 3 to 4 weeks of 
gestation (normal gestation in the sheep is about 
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21 weeks). The advantage for researchers inter- 
ested in questions about prenatal development is 
that such data collection can preserve a record of 
developmental change in real time. But the con- 
comitant disadvantage is that the tremendous vol- 
ume of data forces researchers to adopt explicit 
strategies for sampling, summarizing, and simplify- 
ing the data to discover and extract useful informa- 
tion from it (e.g., Anderson et al., 1998; Robertson 
et al., 1996). And the most severe limitation is that, 
despite the wealth of information provided from a 
large array of behaviorally relevant sensors, it is 
still impossible to draw clear correspondences be- 
tween patterns of change in measured variables 
and actual behavior of the fetus. For instance, see- 
ing a pattern of tongue, mouth, and esophageal ac- 
tivity when the fetus is experimentally exposed to a 
taste-odor cue in the amniotic fluid is sufficient to 
document a behavioral response to a test stimulus, 
but the pattern of measurements must be cali- 
brated to the actual behavior of an observable sub- 
ject (such as a newborn lamb) to conclude that the 
response involved an ingestive or aversion reaction 
to the test stimulus (Robinson et al, 1995). 

Automated tracking of human behavior in the 
field, particularly when arrays of multiple sensors 
are used to record data on a fine time scale, will 
pose similar problems for analysis and interpreta- 
tion. Researchers should plan in advance not only 
how to record sensor data, but how to identify 
practical strategies for summarizing, simplifying, 
or sampling from the collected data set. This task is 
analogous to learning to drink from a fire hose 
(Waldrop, 1990). To do this correctly, it is essential 
to relate patterns of data obtained from recording 
instruments to observable behavior recorded on 
video in controlled environments, as well as to use 
automated tracking to draw inferences about real 
human behavior. 



Data Analysis Strategies 

Automated recording from multiple remote sensors 
offers many possibilities for data analysis that may 
be useful for characterizing both normal and ab- 
normal human behavior. Particularly if sensor data 
are recorded with high temporal resolution, data 
sets will provide more than general summaries of 
activity and can be used to create detailed, quanti- 
tative characterizations of behavior. Time-series 



data (involving any data set that preserves the con- 
tinuous time sequence in which data are recorded) 
can be analyzed using a variety of well-established 
analytic methods to uncover temporal patterns in 
the data. Time-series analyses are particularly use- 
ful for detecting cyclical fluctuations in behavioral 
variables, such as daily (circadian) and higher fre- 
quency (ultradian) rhythmic patterns, and more 
complex patterns that are indicative of dynamic or 
chaotic organization in the data (e.g., Anderson et 
al., 1998; Kelso, 1995; Robertson & Bacher, 1995; 
see figure 8.1). 

Another approach that may simplify large data 
sets is to establish criteria for identifying discrete 
behavioral events from continuous sensor record- 
ings. Remote sensors are likely to produce some 
variable output as noise in the absence of overt be- 
havior, but calibrating these recordings to synchro- 
nized video may help establish thresholds or other 
operational criteria for delineating discrete behav- 
ioral events. For example, continuous EMG data 
from different limb muscles can be simplified by 
characterizing signature patterns associated with 
single extension-flexion cycles of limb movement. 
Thus, a noisy time series that is difficult to com- 
pare between subjects may be collapsed into a 
simplified time series that can be meaningfully 
compared with conventional statistical methods. 
This essentially was the strategy adopted to con- 
dense EMG data from oral and limb muscles into 
movement frequencies to assess behavioral re- 
sponses to experimental stimulation in fetal sheep 
(Robinson etal., 1995). 

Derived time series of categorical events also 
are amenable to other methods for detecting and 
characterizing behavioral organization. One of the 
more powerful of these analytic approaches is the 
Markov analysis of sequential structure from a be- 
havioral data set. A Markov process is defined as a 
sequential structure in which knowledge of a pre- 
ceding category completely specifies the next cate- 
gory of behavior to occur. In data obtained from 
real animals or humans, sequential relationships 
between behavioral categories, or between two or 
more interactants, are rarely this deterministic. 
Rather, sequential structure emerges as a quantita- 
tive set of relations characterized by nonrandom 
probabilities linking successive pairs of events. 
Like time-series approaches, Markov sequential 
analyses can reveal orderliness in the structure of 
ongoing behavior expressed by an individual (e.g., 
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Figure 8.1 . Example of 24-hour rhythmic variation in motor activity derived from continuous time- 
series recordings of EMG from fetal sheep between 120 and 145 days of gestation. Individual movement 
events of a tongue protractor muscle (geniohyoid) were derived and counted in 15-second intervals 
from the EMG data. The time series then was decomposed, extracting a seasonal (24 hr) trend (over 25 
days) and residual components of the time series. The circadian plot identifies the period of dark 
(shaded) and light during the day, and reveals two peaks in oral activity just after lights on and just be- 
fore lights off. Data points depict the mean change in activity per hour relative to overall average activity 
(horizontal line) observed in 18 fetal subjects; vertical bars depict the standard error of the mean (SEM). 



Hailman & Sustare, 1973; Robinson & Smother- 
man, 1992) and in the pattern of communica- 
tive interaction between a parent and child or two 
individuals engaged in dialogue (e.g., Bakeman & 
Gottman, 1997). 



Taxonomies of Tracking Systems 

Mulder (1994) reviewed different configurations for 
recording and classifying human movements that 
are germane to people tracking. Briefly, these systems 
can be classified as inside-inside, inside-outside, and 
outside-inside. Sophisticated systems to record the 
signatures of complex behaviors could be built from 
different combinations of these. 

Inside-inside systems employ sensors and 
sources located on a person's body, such as silver 
chloride electrodes to record surface electromyo- 
graphic activity, eye movements, gastric activity 
(electrogastrogram), galvanic skin response (GSR; 



also referred to as electrodermal response), or a 
glove with piezoelectric transducers to sense defor- 
mation caused by changing configurations of the 
hand. These systems may capture body activity 
while a person roams over a large territory, but may 
be obtrusive, subject to movement artifact, and do 
not provide external verification of behavior and 
correct rejection of artifacts from unanticipated ex- 
ternal sources of noise. 

Inside-outside systems employ sensors on the 
body that detect external sources, either artificial or 
natural. Examples include a scleral coil moving in 
an externally generated electromagnetic field to 
record eye movements, or accelerometers attached 
to the trunk or limbs of a person moving in the 
earth's gravitational field. These systems provide 
some world-based information. For instance, it is 
possible to track distance traveled and energy ex- 
pended from accelerometers attached to a person (as 
in pedometers). However, inside-outside systems 
share some of the same problems as inside-inside 
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systems. Workspace and accuracy are generally lim- 
ited, and there is no external validation of behavior 
unless the system is combined with additional body- 
mounted sensors, such as pinhole cameras or radar 
that show where a person is going. 

Outside-inside systems use external sensors 
that process various sources of information, in- 
cluding images and markers or emitters. Examples 
include optoelectronic systems that track reflective 
markers attached to the body, video tracking of 
reflections from the eye, and global positioning 
system (GPS) data from GPS sensors attached to 
a person. The systems lose data due to occlusion 
when something passes between the person and 
the detector, such as with bifocal spectacles in front 
of some eye trackers and loss of GPS data when 
persons move indoors and walls occlude connec- 
tion to the orbiting satellite. These systems are also 
subject to artifact, depending on the signal source. 
For example, infrared detecting systems are sus- 
ceptible to sunlight. Artificial vision systems that 
track data from optical images in real time have 
huge information processing demands and rely on 
complex algorithms that are highly prone to error 
and artifact. 

Strengths and weaknesses of different tracking 
systems, interfaces, and synchronization between 
different data streams — and the pros and cons of 
different sensors (e.g., optical, audio, electromag- 
netic, mechanical), sensor calibration and drift, and 
data acquisition and analysis software — are impor- 
tant topics for discussion and further research. 



Potential Applications 

A people tracker could be used for locating and 
tracking human activity in a variety of populations 
and circumstances. Such a device might use GPS, 
differential GPS, or arrays of accelerometers on the 
trunk or limb. Different accelerometer implemen- 
tations could include laser/fiber optic, piezoelec- 
tric, and membrane/film. Data from these devices 
might be obtained in synch with other physiologi- 
cal indices (such as heart rate). 

In general, smaller and less obtrusive sensor 
devices are better. Power supply to the devices and 
data storage and downloading are other technical 
issues. Different versions could have a memory for 
collecting data for hours to days to weeks, down- 
loaded remotely (e.g., via cell phone, Internet, 



infrared transmission to a nearby collector, or by 
swapping out a cartridge). A taxonomy would be 
needed to define and set levels for trigger events 
that define activities such as walking, running, 
sleeping, and falling. The signals would need to be 
displayed in an understandable way. 

As these problems are solved, potential neu- 
roergonomic applications can expand, which are 
highly relevant to multiple research areas in healthy 
and impaired individuals. General applications 
in healthy populations could include assessing the 
activity of athletes, hikers, mountaineers, soldiers, 
and medical personnel (nurses, residents, physi- 
cians) and other individuals to assess workforce 
and human ergonomics in a variety of settings, 
including hospitals, factories, war zones, and 
transportation. 

Potential medical applications include assess- 
ing magnitude, progression, or improvement of 
disease. Examples include baseline and longitudi- 
nal assessments of activity and social interactions, 
and of response to interventions in the following: 

1. Advanced age 

2. Depression 

3. Stroke 

4. Attention-deficit/hyperactivity disorder 

5. Multiple sclerosis with remission, relapse, or 
progression 

6. Alzheimer's disease and response to 
cholinesterase inhibitors and other medica- 
tions 

7. Parkinson's disease and response to dopamin- 
ergic agents or deep brain stimulation 

8. Arthritis or fractures affecting the leg, hip, or 
spine, pre- and posttherapy 

9. Alcohol or drug use 

10. Malingering 

1 1 . Vestibular disease and other balance disorders 

12. Pain syndromes (e.g., back pain, migraine, 
fibromyalgia) 

13. Prescription drug use, for example, in cardiac 
disease, pulmonary disease, allergies, insom- 
nia, and hypertension, and so on 

Acute online outputs from these devices might be 
used identify an elderly person who has not shifted 
position for too long due to an illness, or in whom 
an abrupt change in signal suggests a fall to the 
floor. Longitudinal comparisons of chronic data 
output from these devices could be used to track 
disease improvement or progression (e.g., in stroke, 
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Alzheimer's disease, Parkinsons disease), and objec- 
tively assess outcomes of interventions (drugs, re- 
habilitation) in clinical trials. 



Tracking Human Movement 
and Energy Expenditure 

Early efforts on tracking human activity, outside of 
simple photography and video analysis, focused on 
the activity of children and adults and were based 
mostly on accelerometer ouputs. There is a body 
of work in exercise physiology on tracking energy 
expenditure in behaving humans (Welk, 2002). 
Researchers may calibrate instruments to energy 
expenditure and use Ainsworth's Physical Activity 
Compendium for organizing data (http://preven 
tion.sph.sc.edu/Tools/compendium.htm). In some 
situations, output is calibrated to observation using 
observational systems such as the Children's Activ- 
ity Rating Scale (e.g., Puhl, Greaves, Hoyt, & Bara- 
nowski, 1990) or the System for Observing Play 
and Leisure Activity in Youth of McKenzie, Mar- 
shall, Sallis, and Conway (2000). 

Researchers assess the integrity of accelerome- 
ters in advance of use by testing their output before 
and after subjecting them to vigorous mechanical 
shaking. Procedures for calibrating accelerometers 
in pedometers strapped around the ankles involve 
the subject walking on a treadmill with a certain 
stride length and speed. Accelerometers can be 
placed in different locations and combinations on 
the body. Axial accelerometers can be placed on the 
hip as a reflection of movement of the main body 
mass. Accelerometers can be placed on the limbs 
to assess arm and leg movement, for example, to 
compare the swing of paretic and unimpaired sides 
during walking. For greater detail on limb coordi- 
nation, it may be necessary to place at least two 
sensors on each limb. 

Accelerometry can be combined with other 
physical measures such as skin temperature, sweat- 
ing, heart rate, and so on, to give a more complete 
profile of what a person is doing. Wearable devices 
can be put into clothing, for example, accelerome- 
ters in pants pockets and heart sensors in bras. 
Contemporaneous estimates of energy expenditure 
using radiolabeled water can help provide an inde- 
pendent index of metabolic activity (Levine et al, 
2005). Estimates of activity based on self-report are 
often inaccurate. 



Levine et al. (2005) used a Tracmor triaxial ac- 
celerometer system (Maastricht, Netherlands), a 
validated system for detecting body movement in 
free-living subjects, to assess changes in posture 
and movement associated with the routines of 
daily life known as nonexercise activity thermogen- 
esis. Results of this pilot study showed that activi- 
ties that include getting up to stretch, walking to 
the refrigerator, and just plain fidgeting are greater 
in lean people than in sedentary people, account 
for hundreds of calories of energy expenditure per 
day, and may make the difference between being 
lean and obese. 

An application of movement sensors is to 
gauge the activity of children with attention-deficit/ 
hyperactivity disorder. There can be many other 
applications in normal and in cognitively impaired 
individuals (Jovanov, Milenkovic, Otto, & deGroen, 
2005; Sieminski, Cowell, Montgomery, Pillai, & 
Gardner, 1997). Macko et al. (2002, 2005) moni- 
tored step activity of hemiparetic stroke patients 
moving freely. Technical issues included the algo- 
rithm for identifying slow gait (<0.5 mph), in- 
tegrating physiological monitoring of multiple 
variables, battery life, ease of setup, and obtrusive - 
ness and comfort of the instrumentation package 
to be worn. The Cyma Corp StepWatch showed 
acceptable accuracy, reliability, and clinical utility 
for step monitoring in stroke patients. Validated 
step activity might be expanded to coregister time- 
marked heart rate, GPS position, and other mea- 
sures. Multiple devices might be used to assess both 
social and physiological interactions in circum- 
stances such as crowds, parties, pedestrian traffic, 
and so on, with lessons from ethology. 

Benbadis, Siegrist, Tatum, Heriaud, and An- 
thony (2004) found that short-term outpatient 
EEG video monitoring could provide evidence to 
discriminate between psychogenic (nonepileptic) 
seizures and true epileptic seizures in patients with 
atypical spells. In this case, the results from a sim- 
ple combination of sensors eliminated the need for 
expensive inpatient EEG video monitoring. The re- 
sults helped avoid improper drug administration in 
patients with alternative diagnoses that require a 
different approach, such as with cardiac or vestibu- 
lar symptoms, anxiety attacks, somatization disor- 
der, or malingering. 

Accelerometers combined with light sensors 
may help tell if a person has gone outside or stayed 
inside during an activity. Combination with video 
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data can help identify a specific environment and 
activity a person is performing. Some accelerome- 
ters can be triggered with a bar magnet to mark 
when a certain kind of behavior occurs. A synchro- 
nously triggered voice recorder can document a 
person's narrative of what may have just occurred. 
In combination with several of the neuroergonom- 
ics techniques described in part II of this volume, 
the setups have the potential to answer many ques- 
tions in human factors research and medicine. 
What factors trigger a stroke patient to move a neg- 
lected or paretic arm, or a person with Parkinsons 
disease to start walking? What behaviors or internal 
states precede an anxiety attack, falling asleep while 
piloting a plane or driving a car, an industrial acci- 
dent, or a medical error by health care personnel? 



Tracking over Long Distances 

Human behavior can be studied as people travel 
afoot. It can also be studied as people travel over 
larger distances (see chapter 9, this volume). This 
can be done using instrumented vehicles (IVs). In- 
ternal networks of modern vehicles make it possi- 
ble to obtain detailed information from the drivers 
own car (Rizzo, Jermeland, & Severson, 2002). 
Modern vehicles report certain variables relevant to 
speed, emissions controls, and vehicle perfor- 
mance, and even seatbelt and headlight use, cli- 
mate and traction control, wheel speed, and 
antilock braking system (ABS) activation. In addi- 
tion, lane-tracking video can be processed with 
computer algorithms to assess lane-keeping behav- 
ior. Radar systems installed in the vehicle can 
gather information on the proximity, following dis- 
tance, and lane-merging behavior of the driver and 
other neighboring vehicles on the road. GPS sys- 
tems can show where and when a driver drives, 
takes risks, and commits safety errors. Wireless 
systems can check the instrumentation and send 
performance data to remote locations. Together, 
these developments can provide direct, real-time 
information on driver strategy, vehicle usage, up- 
keep, drive lengths, route choices, and decisions to 
drive during inclement weather and high traffic 
that cannot be observed any other way. Continuous 
data obtained in these IVs provide key information 
for traffic safety research and interventions and are 
highly relevant to the larger issues of studying hu- 
man strategies, tactics and cognitive errors in the 



real world, in natural situations, in which people 
behave like they do in real settings. 

Multiple studies have used IVs in traffic safety 
research (e.g., Dingus et al., 2002; Fancher et al., 
1998; Hanowski, Wierwille, Garness, & Dingus, 
2000). In most cases an experimenter is present, 
and drivers who are aware of being observed are li- 
able to drive in an overly cautious and unnatural 
manner. Because total data collection times are 
often less than an hour and crashes and serious 
safety errors are relatively uncommon, no study 
until recently has captured precrash or crash data 
for a police-reported crash, and no information is 
available on general vehicle usage. Instead, insights 
on vehicle usage by at-risk drivers have relied on 
questionnaires completed by individuals who may 
have defective memory and cognition. 

The main data that transportation researchers 
have on actual collisions and contributing factors 
are collected post hoc (in forensic or epidemiologi- 
cal research). These data are highly dependent 
upon eyewitness testimony, driver memory, and 
police reports, all of which have serious limita- 
tions. The best information we have had regarding 
near collisions in at-risk drivers comprises anec- 
dotal reports by driving evaluators and instructors 
(usually testing novice drivers) and police reports 
of moving violations. Most of these potential crash 
precursors, if they are ever even recognized at all, 
are known only to the involved parties and are 
never available for further study and subsequent 
dissemination of safety lessons. 

A driver driving his or her own IV is exposed 
to the usual risk of the real-world road environ- 
ment that he or she is normally exposed to, with- 
out the psychological pressure that may be present 
when a driving evaluator is in the car. Road test 
conditions can vary depending on the weather, 
daylight, traffic, and driving course. However, this 
is an advantage in naturalistic testing, where re- 
peated observations in varying real-life settings can 
provide a wealth of information regarding driver 
risk acceptance, safety countermeasures, and adap- 
tive behaviors, and unique insights on the ranging 
relationships between low-frequency, high-severity 
driving errors and high-frequency, low-severity 
driver errors. These types of brain-in-the-wild rela- 
tionships were explored in detail in a Virginia Tech 
Transportation Institute/National Highway Trans- 
portation Safety Administration study of driving 
performance and safety errors in 100 neurologically 



The Brain in the Wild 



121 



normal individuals, driving 100 total driver years 
(Dingus et al., 2006; Neale, Dingus, Klauer, Sud- 
weeks, & Goodman, 2005). 

Hidden sensors detected vehicle longitudinal 
acceleration and rate of yaw (lateral acceleration). 
Infrared sensors detected cell phone use. Readings 
off the internal network of each IV provided infor- 
mation on speed, use of controls, seatbelt and 
headlight use, traction control, wheel speed, and 
ABS activation, and air bag deployment. GPS out- 
put showed where and when a driver drove. Five 
miniature charge coupled devices (CCD) cameras 
mounted in the vehicle provided video information 
of the (1) driver's face plus driver's side of vehicle, 
(2) forward view, (3) instrument panel, (4) passen- 
ger side of vehicle, and (5) rear of vehicle. 

A data acquisition system onboard each IV con- 
tinuously stored measured variable data and video 
streams from five cameras during driving sessions. 



The internal hard drive had the capacity to store at 
least 4 weeks of 6-hour-long driving days. Each IV 
had a cell phone account to enable the investigators 
to download video and vehicle data "snippets" to 
verify proper systems operation, query the data 
system for GPS location information, and delete 
archived data from the vehicle hard drive. A chase 
vehicle allowed investigators to troubleshoot the ex- 
perimental IVs and to download raw data from the 
IVs (at least once every 2 weeks). 

Raw data obtained from each IV were filtered 
using specific criteria to flag where a critical inci- 
dent may have occurred in the IV data stream. For 
example, longitudinal and lateral accelero meters 
measured g-forces as drivers applied the brakes or 
swerve to miss an obstacle, and these were used to 
flag critical driving situations in the data stream. 
Specific values (indicated by X in table 8.1) for 
each of these variables were used to determine when 



Table 8.1. Trigger Criteria to Flag Critical Incidents 



Trigger Type 



Description 



1. Lateral acceleration 

2. Longitudinal 
acceleration 

3. Lane deviation 

4. Normalized 
lane position 

5. Forward time 
to collision 

6. Rear time to 
collision 

7. Yaw rate 

8. ABS brake status 

9. Traction control 



10. Airbag status 

11. RF sensor 

12. Seatbelt 



Lateral motion equal or greater than 0.X g.* Will indicate when a driver has 

swerved to miss an obstacle. 

Acceleration or deceleration equal to or greater 

than 0.X g. Will indicate when a driver has either braked hard or accelerated 

hard to avoid an obstacle. 

Activated if the driver crosses the solid line border (Boolean occurrence). May 

indicate when a driver is either inattentive or losing control of the vehicle. 

Activated if the driver's path deviates by X.X% of 

centerline. May indicate if a driver is inattentive or losing control of the vehicle. 

Activated if the driver followed the preceding 

vehicle at X range/range-rate. May indicate if a driver is following another 

vehicle too closely and/or demonstrating unsafe or aggressive driving. 

Activated if the driver following the IV is closing 

in on the IV at a rate of X range/range-rate. May indicate if a driver is being 

followed too closely. 

Activated if the lateral motion of the vehicle is 0.X g. Will be an indication if a 

driver has swerved or is rapidly turning the steering wheel. 

Activated if the ABS brakes are active. Note: only applicable to those vehicles 

that have ABS brakes. Will provide another indication of when a driver is 

braking hard. 

Activated if the traction control system comes on. Note: only applicable to 

those vehicles that have traction control and can be monitored via the in- 

vehicle network. May indicate when a driver may potentially lose control of the 

vehicle. 

Activated if the airbag is deployed. Will indicate a collision has occurred. 

Activated if the driver is using a cell phone or a PDA when the vehicle is on. 

May indicate when the driver is distracted. 

Activated when car is in motion and seat belt is not fastened. 



*Specific values (X) for each of these variables are used Co determine possible critical incidents. 
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Optimized Trigger 

Goal: Minimize False Alarms 



Liberal Trigger 

Goal: Minimize Misses 




Figure 8.2. Graphical depiction of where the trigger criteria will be set to minimize misses and false 
alarms in classification of critical incidents. 



a possible critical incident occurred. A sensitivity 
analysis was performed by setting the trigger crite- 
ria to a liberal level (figure 8.2, right side) to reduce 
the chance of a missed valid incident while allow- 
ing a high number of invalid incidents (false 
alarms). 

As mentioned above, critical incident triggers 
(e.g., swerving, rapid breaking) cause the auto- 
matic flagging of that segment of the IV data 
stream. These flags initiate standardized review of 
the relevant, time-linked IV data collected immedi- 
ately preceding and following the critical incident. 
The reviews are based on objective decision trees, 
which culminate in a taxonomy of driver errors 
and decisions. These classification schemata were 
developed to classify the myriad events that occur 
with large-scale traffic observations (e.g., Wierwille 
etal.,2002). 

This framework allows objective and detailed 
analysis of preincident maneuvers, precipitating 
factors, and contributing factors, far more than is 
possible with epidemiological (postincident) stud- 
ies. The flagged segments of the IV data stream are 
analyzed to determine the sequence of events sur- 
rounding each critical incident, allowing character- 
ization of the following: 

• Driver actions just prior to onset of the inci- 
dent 

• Type of incident that occurred 

• Precipitating factor (initial action that started 
the incident sequence) 



• Causal factors or any action or behavior that 
contributed to the outcome of the incident 

• Type of evasive maneuvers performed (if any) 

By applying the data reduction schemata listed 
above, critical incidents are classified into either (a) 
appropriate responses to an unsafe event, or (b) the 
three basic categories of inappropriate responses 
that serve as dependent measures in this study: 

• Driver errors: Driver safety errors in the 
absence of a nearby object. 

• Near crashes (close calls or near misses): 
Events that require a rapid evasive maneuver 
but no physical contact is made with an object 
(person, vehicle, guardrail, etc.). 

• Crashes: Incidents in which physical contact is 
made. 

Examples of the steps in the application of 
these classification tools to flagged portions of the 
data stream from the IV are shown in figures 8.3 
and 8.4. Application of the tree in figure 8.3 could 
result in many possible outcomes, one of which is 
"Conflict with a lead vehicle." Figure 8.4 shows 
further classification of "Conflict with lead vehicle" 
in the case of a crash. 

Data collection in the 100-Car Naturalistic Dri- 
ving Study was completed in 2004, and the final 
results of the study were recently compiled. All en- 
rolled drivers allowed installation of an instrumenta- 
tion package into their own vehicle (78 cars) or to 
use a new model-year IV provided free of charge for 
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Determine the nature of traffic 

conflict for trigger/epoch (note: 

may have more than one traffic 

conflict per epoch) 

I 

'Identify all other vehicles in near^ 

proximity and assign a numerical 

identifier (Note: the number of 

vehicles assigned will determine how 

many vehicles must be reduced), 

"X" 



Note Driver A 
Number J 



l=Conflict with a lead vehicle 
2=Conflict with a following vehicle 
3=Conflict with oncoming traffic 
4=Conflict with vehicle in adjacent lane 
5=Conflict with merging vehicle 
6=Conflict with vehicle turning across 
subject vehicle path (same direction) 
7=Conflict with vehicle turning across 
subject vehicle path (opposite direction) 
8=Conflict with vehicle turning into 
subject vehicle path (same direction) 
9=Conflict with vehicle turning into 
subject vehicle path (opposite direction) 
10=Conflict with vehicle moving across 
subject vehicle path (though intersection) 
1 l=Conflict with parked vehicle 
12=Conflict with pedestrian 
f 3=Conflict with pedalcyclist 
14=Conflict with animal 
15=Conflict with obstacle/object in roadway 
f 6=Single vehicle conflict 
18=Other (specify) 

19=No known conflict (for RF sensor trigger) 
99=Unknown conflict 



Figure 8.3. Depiction of sequence of steps used to locate and classify triggered incidents in the IV data stream. 
From these incidents, multiple types of traffic conflicts are identified for further analyses, as in figure 8.4. 
CL = conflicts, NC = no conflicts. 



their own use (22 cars). The average age of the pri- 
mary drivers in the study was 36, with 61% being 
male. Data collection for the 100-Car Study repre- 
sented monitoring periods of 12-13 months per ve- 
hicle, resulting in almost 43,000 hours of actual 
driving data, and approximately 2,000,000 vehicle 
miles. Overall, there were a total of 69 crashes (see 
figure 8.5), 761 near crashes, and 7,479 other rele- 
vant incidents (including 5,568 driver errors) for 
which data could be completely reduced (see figure 
8.4). The severity of crashes varied, with 75% being 



mild impacts, such as when tires strike curbs or 
other obstacles. Using taxonomy tools to classify all 
relevant incidents (see figures), the majority of inci- 
dents could be described as "lead vehicle"; however, 
several other types of conflicts (adjacent vehicle, 
following vehicle, single vehicle, object/obstacle) 
also occurred at least 100 times each. 

The next phase of this work will examine how 
cognitive impairments are related to driver errors 
and crashes. Analyses of the IV data stream can 
show specific driver behaviors that led to critical 
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Figure 8.4. Decision tree (partial) for assessing a critical incident: conflict with lead vehicle (traffic conflict type #1). 
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Figure 8.5. A real-life crash produces an abrupt longitudinal deceleration in the electronic record from the IV 
GPS data show the exact location of the crash on the map. Corresponding video data (the quad view) show the 
driver was looking down (to eat a sandwich) and reacted too late to a braking lead vehicle. 



incidents in impaired drivers. For example, longitu- 
dinal accelerations combined with braking behavior 
or lateral acceleration may indicate a driver reacted 
abruptly to avoid an obstacle, for example, because 
of an unsafe go/no-go decision to pass through 
an intersection or merge between lanes. Lane de- 
viations and lateral accelerations may indicate 
executive misallocation of attention, as in drivers 
whose "eyes off-road" time increases (Dingus, 1995; 
Zwahlen & Balasubramanian, 1974). Critical inci- 
dents identified in the IV data stream could also be 
interpreted with respect to differing ambient con- 
ditions. Drivers traveling on different roads and 
at different times of the year face different safety 
contingencies, depending on traffic, atmospheric 
conditions, and time of day. However, a critical con- 
ceptual issue with respect to decision making is 
how drivers respond to safety contingencies, re- 



gardless of when or where they arise. GPS and 
video data for the IV of each decision-making- 
impaired driver can be linked to contemporaneous 
data from National Weather Service and Department 
of Transportation sites and archives. This allows 
monitoring of driver-specific road and weather con- 
ditions that coincide with the IV performance data 
and number of trips, trip lengths, total miles driven, 
and speed in unfavorable weather and lighting. Fail- 
ure to adjust to altered contingencies is a hallmark 
of decision-making impairment due to brain lesions 
and is the focus of new naturalistic studies of 
driving. Table 8.2 lists examples of potential rela- 
tionships between cognitive tests, the executive 
functions and constructs measured by these tests, 
and unsafe driver behaviors associated with these 
functions, and suggests possible answers to the 
question, "What if Phineas Gage could drive?" 



126 



Neuroergonomics Methods 



Table 8.2. Off-Road Tests and the Measure Functions/Constructs That Predict Unsafe Driver 
Behaviors That May Lead to a Near Crash or Crash 



Test 



Function/Construct 
Measured 



Examples of Unsafe Driver 
Behaviors Associated 
with Impaired Functions 



WCST 



Response to changing 
contingencies 



Trails B 
Trails A 


Response alternation 


Tower of Hanoi 
Stroop 


Planning and execution 
of multistep tasks 
Susceptibility to 
interference 


Gambling 


Decision making 


Go/no-go 


Impulse control 


Digit span 


Working memory 



Failure to adjust speed 

or following distance to changing road conditions 
Failure to alternate eye 

gaze appropriately between road, mirrors, and gauges 
Sudden brake application; 
swerving across lane; running car near empty 
Glances of >2 s off road; 
e.g., with passenger present or while eating 
Traffic violation, e.g., speeding and other traffic viola- 
tions; engaging in behavior extraneous to driving 
Running red light; engaging in behavior extraneous to 
driving 

Cutting off vehicles because of forgetting their location; 
disregard for following vehicle; driving slowly in left lane 



Note. Occurrence of the unsafe behavior is assessed from [lagged segments of the IV data stream. Trails = Trail-Making Test, 
Parts A and B; WCST — Wisconsin Card-Sorting Task. 



Conclusion 

Modern technology allows development of various 
"people trackers" using combinations of accelerom- 
eters, GPS, video, and other sensors (e.g., to mea- 
sure cerebral activity, eye movement, heart rate, 
skin temperature) to make naturalistic observations 
of human movement and behavior. These devices 
can advance the goal of examining human perfor- 
mance, strategies, tactics, interactions, and errors in 
humans engaged in real-world tasks. Besides vari- 
ous issues of device development and sensor choice 
and placement, we need to develop taxonomies for 
classifying likely behavior from sensor output. We 
also need to be able to analyze behavior sequences 
using new applications of classic ethological tech- 
niques. Different implementations can provide 
unique data on how the brain interacts with diverse 
environments and systems, at work and at play, and 
in health, fatigue, or disease states. 



Main Points 

1 . People often act differently in the real world 
than they or their relatives indicate. Con- 
sequently, it is important to consider more 



direct sources of evidence of human behavior 
in natural and naturalistic settings. 
Modern technology allows development of 
various "people trackers" using combinations 
of accelerometers, GPS, video, and other 
sensors (e.g., to measure cerebral activity, eye 
movement, heart rate, skin temperature) to 
make naturalistic observations of human 
movement and behavior. 
These devices can assess physiology and 
behavior from different vantages (such as 
outside looking inside and inside looking 
outside) to evaluate the behavior of people 
who are seeing, feeling, attending, deciding, 
erring, and self-correcting during activities of 
daily living. 

Analysis of behavior sequences measured with 
these devices can draw from classic ethological 
techniques. 

Current challenges for device development 
concern sensor choice and placement and 
development of taxonomies for classifying 
behavior from sensor output. 
Different device implementations can provide 
unique data on how the brain interacts with 
diverse environments and systems, at work and 
at play, and in healthy and impaired states. 



The Brain in the Wild 



127 



Key Readings 

Dingus, T. A., Klauer, S. G., Neale, V L., Petersen, A., 
Lee, S. E., Sudweeks, J., et al. (2006). The 100-Car 
Naturalistic Driving Study: Phase II — Results of the 
100-car field experiment (Project Report for 
DTNH22-00-C-07007, Task Order 6; Report No. 
TBD). Washington, DC: National Highway Traffic 
Safety Administration. 

Lehner, P N. (1996). Handbook of ethological methods (2nd 
ed.). Cambridge, UK: Cambridge University Press. 

Levine.J. A., Lanningham-Foste, L. M., McCrady S. K., 
Krizan, A. C, Olson, L. R., Kane, P. H., et al. 
(2005). Interindividual variation in posture alloca- 
tion: Possible role in human obesity science. Sci- 
ence, 307, 584-586. 

Robinson, S. R., Wong, C. H., Robertson, S. S., 

Nathanielsz, P W, & Smotherman, W P. (1995). 
Behavioral responses of a chronically-instrumented 
sheep fetus to chemosensory stimuli presented in 
utero. Behavioral Neuroscience, 109, 551-562. 



References 

Anderson, C. M., Mandell, A. J., Selz, K. A., Terry, 
L M., Wong, C. H., Robinson, S. R., et al. (1998). 
The development of nuchal atonia associated with 
active (REM) sleep in fetal sheep: Presence of re- 
current fractal organization. Brain Research, 787, 
351-357. 

Bakeman, R., & Gottman, J. M. (1997). Observing inter- 
action: An introduction to sequential analysis (2nd 
ed.). Cambridge: Cambridge University Press. 

Benbadis, S. R., Siegrist, K, Tatum, W O., Heriaud, L., 
& Anthony, K. (2004). Short-term outpatient EEG 
video with induction in the diagnosis of psy- 
chogenic seizures. Neurology, 63(9), 1728-1730. 

Benton, A. L. (1991). The prefrontal region: Its early his- 
tory. In H. S. Levin, H. M. Eisenberg, & A. L. Ben- 
ton (Eds.), Frontal lobe junction and dysfunction 
(pp. 3-12). New York: Oxford University Press. 

Damasio, A. R. (1996). The somatic marker hypothesis 
and the possible functions of the prefrontal cortex. 
Philosophical Transactions of the Royal Society of Lon- 
don (Biology), 351, 1413-1420. 

Damasio, A. R. (1999). The feeling of what happens: Body 
and emotion in the making of consciousness . New 
York: Harcourt Brace. 

Damasio, H., Grabowski, T, Frank, R., Galaburda, 
A. M., & Damasio, A. R. (1994). The return of 
Phineas Gage: Clues about the brain from the skull 
of a famous patient. Science, 264, 1102-1105. 

Dingus, T. A. (1995). Moving from measures of perfor- 
mance (MOPS) to measures of effectiveness 



(MOEs) in the safety evaluation of ITS products or 
demonstrations. In D. Nelson (Ed.), Proceedings of 
the ITS Safety Evaluation Workshop. Washington, 
DC: ITS America. 

Dingus, T. A., Klauer, S. G, Neale, V L, Petersen, A., 
Lee, S. E., Sudweeks, J., et al. (2006). The 100-Car 
Naturalistic Driving Study: Phase II — Results of the 
100-car field experiment (Project Report for 
DTNH22-00-C-07007, Task Order 6; Report No. 
TBD). Washington, DC: National Highway Traffic 
Safety Administration. 

Dingus, T A., Neale, V L., Garness, S. A., Hanowski, R., 
Keisler, A., Lee, S., et al. (2002). Impact of sleeper 
berth usage on driver fatigue. Final Project Report. 
(Report No. 61-96-00068). Washington, DC: U.S. 
Department of Transportation, Federal Motor Car- 
riers Safety Administration. 

Fancher, P, Ervin, R., Sayer, J., et al. (1998). Intelligent 
cruise control field operational test: Final report (Re- 
port No. DOT-HS-808-849). Washington, DC: 
U.S. Department of Transportation, National High- 
way Traffic Safety Administration. 

Hailman, J. P, & Sustare, B. D. (1973). What a 
stuffed toy tells a stuffed shirt. Bioscience, 23, 
644-651. 

Hanowski, R. J., Wierwille, W W, Garness, S. A., & 
Dingus, T A. (2000). Impact of local/short haul op- 
erations on driver fatigue: Final report (Report No. 
DOT-MC-00-203). Washington, DC: U.S. Depart- 
ment of Transportation, Federal Motor Carriers 
Safety Administration. 

Jarvis, R., & Janz, K. (2005). An assessment of daily 
physical activity in individuals with chronic heart 
failure. Medicine and Science in Sports and Exercise, 
37(5, Suppl.), S323-S324. 

Jovanov, E., Milenkovic, A., Otto, C, & deGroen, P. C. 
(2005). A wireless body area network of intelligent 
motion sensors for computer assisted physical re- 
habilitation. Journal of NeuroEngineering and Reha- 
bilitation, 2, 6. 

Kelso, J. A. S. (1995). Dynamic patterns: The self-organi- 
zation of brain and behavior. Cambridge, MA: MIT 
Press. 

Lehner, P N. (1996). Handbook of ethological methods 
(2nd ed.). Cambridge, UK: Cambridge University 
Press. 

Levine.J. A., Lanningham-Foste, L. M., McCrady, S. K, 
Krizan, A. C, Olson, L. R., Kane, P. H., et al. 
(2005). Interindividual variation in posture alloca- 
tion: Possible role in human obesity science. Sci- 
ence, 307, 584-586. 

Macko, R. F, Haeuber, E., Shaughnessy M., Coleman, 
K. L., Boone, D. A., Smith, G. V, et al. (2002). 
Microprocessor-based ambulatory activity moni- 
toring in stroke patients. Medicine and Science in 
Sports and Exercise, 34, 394-399. 



128 



Neuroergonomics Methods 



Macko, R. E, Ivey, E M., Forrester, L. W, Hanley, D., 
Sorkm, J. D., Katzel, L. I., et al. (2005). Treadmill 
exercise rehabilitation improved ambulatory func- 
tion and cardiovascular fitness in patients with 
chronic stroke. A randomized, controlled trial. 
Stroke, 36, 2206-2211. 

McKenzie, T. L., Marshall, S. J., Sallis, J. E, & Conway, 
T. L. (2000). Leisure-time physical activity in 
school environments: An observational study us- 
ing SOPLAY. Preventive Medicine, 30, 70-77. 

Mulder, S. (1994, July). Human movement tracking 
technology. Hand-Centered Studies of Human 
Movement Project, Simon Fraser University. Tech- 
nical Report 94-1. 

Neale, V L., Dingus, T. A., Klauer, S. G, Sudweeks, J., 
& Goodman, M. J. (2005). An overview of the 
100-car naturalistic study and findings. Interna- 
tional Technical Conference on the Enhanced Safety of 
Vehicles (CD-ROM). Washington, DC: National 
Highway Traffic Safety Administration. 

Prete, E R. (Ed.). (2004). Complex worlds from simpler 
nervous systems. Cambridge, MA: Bradford MIT 
Press. 

Puhl, J., Greaves, K., Hoyt, M., & Baranowski, T 
(1990). Children's Activity Rating Scale (CARS): 
Description and calibration. Research Quarterly for 
Exercise and Sport, 61(1), 26-36. 

Rizzo, M., Jermeland, J., & Severson, J. (2002). Instru- 
mented vehicles and driving simulators. Gerontech- 
nology, 1, 291-296. 

Robertson, S. S., & Bacher, L. E (1995). Oscillation and 
chaos in fetal motor activity . In J. P. Lecanuet, N. A. 
Krasnegor, W P Fifer, & W P Smotherman (Eds.), 
Fetal development: A psychobiological perspective (pp. 
169-189). Hillsdale, NJ: Erlbaum. 

Robertson, S. S., Johnson, S. L., Bacher, L. E, Wood, 
J. R., Wong, C. H, Robinson, S. R., et al. (1996). 
Contractile activity of the uterus prior to labor 
alters the temporal organization of spontaneous 
motor activity in the fetal sheep. Developmental 
Psychobiology, 29, 667-683. 



Robinson, S. R., & Smotherman, W P (1992). The 
emergence of behavioral regulation during fetal de- 
velopment. Annals of the New York Academy of Sci- 
ences, 662, 53-83. 

Robinson, S. R., Wong, C. H, Robertson, S. S., 

Nathanielsz, P W, & Smotherman, W P. (1995). 
Behavioral responses of a chronically-instrumented 
sheep fetus to chemosensory stimuli presented in 
utero. Behavioral Neuroscience, 109, 551-562. 

Rolls, E. T. (1999). The brain and emotion. Oxford, UK: 
Oxford University Press. 

Rolls, E. T. (2000). The orbitofrontal cortex and re- 
ward. Cerebral Cortex, 10, 284-294. 

Sieminski, D. J., Cowell, L. L., Montgomery, P S., Pil- 
lai, S. B., & Gardner, A. W (1997). Physical activ- 
ity monitoring in patients with peripheral arterial 
occlusive disease. Journal of Cardiopulmonary Reha- 
bilitation, 17(1), 43-47. 

Towell, M. E., Figueroa, J., Markowitz, S., Elias, B., & 
Nathanielsz, E (1987). The effect of mild hypox- 
emia maintained for twenty-four hours on mater- 
nal and fetal glucose, lactate, Cortisol, and arginine 
vasopressin in pregnant sheep at 122 to 139 days' 
gestation. American Journal of Obstetrics and 
Gynecology, 157, 1550-1557. 

Waldrop, M. M. (1990). Learning to drink from a fire 
hose. Science, 248, 674-675. 

Welk, G. (Ed.). (2002). Physical activity assessmentfor 
health-related research. Champaign, IL: Human 
Kinetics. 

Wierwille, W W, Hanowski, R. J., Hankey J. M., 
Kieliszewski, C. A., Lee, S. E., Medina, A., et al. 
(2002). Identification and evaluation of driver errors: 
Overview and recommendations (Final Report for 
Federal Highway Administration contract DTFH 
61-97-C-00051). Washington, DC: Federal High- 
way Administration. 

Zwahlen, H. T, & Balasubramanian, K. N. (1974). A the- 
oretical and experimental investigation of automo- 
bile path deviation when driver steers with no visual 
input. Transportation Research Record, 520, 25-37. 




Perception, Cognition, 
and Emotion 



This page intentionally left blank 




Eleanor A. Maguire 



Spatial Navigation 



Space: Ubiquitous Yet Elusive 

We are always somewhere. "Our body occupies 
space, it moves through space, it interacts with 
things in space, and it can mentally rotate and ma- 
nipulate representations of space. Other objects 
also occupy space and maintain relations in space 
with one another and with us" (Kolb & Whishaw, 
1990, p. 643). As humans, our ability to operate in 
large-scale space has been crucial to our adaptation 
and survival. Even now, a sizeable chunk of our 
day is spent trying to get from place to place, 
whether it is work, home, school, or the store. 
Many of us have experienced navigation-related ar- 
guments on a road trip, the annoyance of taking 
the wrong route, punctuating our cell phone con- 
versations with inquiries about the other party's 
location, and the city dwellers among us are bom- 
barded with information about traffic flow and what 
routes to avoid. We are a species constantly on the 
move. 

The significance of our spatial abilities is most 
poignantly revealed when they become impaired. 
Patients who, because of brain injury or disease, 
are unable find their way in large-scale space can 
experience a devastating loss of independence and 
social isolation (Aguirre & D'Esposito, 1999; Bar- 



rash, 1998; Maguire, 2001). An even more funda- 
mental role for space has also been proposed. 
While spatial navigation is a cross-species behav- 
ior, it has been suggested that in humans this abil- 
ity has evolved into the basic scaffolding for 
episodic memory (Burgess, Maguire, & O'Keefe, 
2002; O'Keefe & Nadel, 1978; but see Eichen- 
baum, 2004). The life events that comprise our in- 
dividual personal history always have a spatial 
context, and when patients suffer dense amnesia, 
it often co-occurs with navigation deficits (Spiers, 
Burgess, Hartley, Vargha-Khadem, & O'Keefe, 2001). 
Given that large-scale space is the backdrop for all 
behavior we direct toward the external world, and 
that it may be fundamental to our internal memory 
representations, how does the human brain sup- 
port our complex yet apparently seamless naviga- 
tion behavior? 

The study of spatial memory has a long and 
productive history. A common experimental ap- 
proach in cognitive psychology and neuropsychol- 
ogy has been to extrapolate from simplified spatial 
stimuli studied in a laboratory setting. In this way, 
confounding variables are minimized and experi- 
mental control maximized, making it possible to 
examine relatively pure cognitive processes. How- 
ever, performance on laboratory tasks has been 
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found to dissociate from actual wayfinding ability 
in real environments (e.g., Nadolne & Stringer, 
2001). Thus, there are clearly factors less amenable 
to examination using standard laboratory tasks re- 
quiring a complementary approach utilizing real- 
world or ecologically valid paradigms (Bartlett, 
1932). That said, using naturalistic tasks presents 
significant challenges in terms of experimental con- 
trol and data interpretation. However, in other dis- 
ciplines, real large-scale environments have been 
studied. Geographers and urban planners have long 
examined different components of large-scale space 
such as landmarks, paths, and districts (Lynch, 
1960). Environmental psychologists have studied 
the factors affecting wayfinding in complex spaces 
(Downs & Stea, 1973; Evans, 1980). Developmen- 
tal psychologists have theorized about how spatial 
knowledge is acquired and its key determinants 
(Hermer & Spelke, 1994; Siegal & White, 1975). 
Meanwhile, animal physiologists have identified 
neurons (place cells) in a part of the rat brain called 
the hippocampus that exhibited location-specific 
firing (O'Keefe & Dostrovsky, 1971; O'Keefe & 
Nadel, 1978). In neuropsychology, as well, patients 
with a variety of etiologies have been reported with 
wayfinding deficits in the real world (Aguirre & 
D'Esposito, 1999; Barrash, 1998; Maguire, 2001; 
Uc, Rizzo, Anderson, Shi, & Dawson, 2004). 

Until about 10 years ago, these different 
strands of spatial research were largely singular in 
pursuing their own particular interests, and this 
was not surprising. It was acknowledged that at- 
tempts should be made to consider how the navi- 
gating brain interacted with its natural context, 
namely the real world (Nadel, 1991; O'Keefe & 
Nadel, 1978). However, the means to achieve this 
was lacking, with no way to map functions onto 
specific human brain regions, or to test navigation 
in truly naturalistic environments while maintain- 
ing some degree of experimental control. Two 
breakthroughs in the last 10 years have begun to 
facilitate an interdisciplinary approach to spatial 
navigation. The first of these was the development 
of brain imaging technologies, particularly mag- 
netic resonance imaging (MRI). Not only does MRI 
produce high-resolution structural brain images, 
but use of techniques such as echo planar imaging 
(EP1) permits inferences about changes in neural 
activity, making human cognition accessible in 
vivo. The use of functional brain imaging to study 
navigation is not without its problems, however. 



Most obvious is how to get participants to navigate 
while their heads are fixed in a physically restricted 
and noisy brain scanner. Methods used to cir- 
cumvent this have included the use of static pho- 
tographs of landmarks and scenes (Epstein & 
Kanwisher, 1998), having subjects mentally navi- 
gate during scanning (Ghaem et al., 1997; Maguire, 
Frackowiak, & Frith, 1997), or showing films of 
navigation through environments (Maguire, Frack- 
owiak, & Frith, 1996). While insights into the 
neural basis of aspects of spatial memory have cer- 
tainly been gained from such studies, clearly the 
optimal navigation task during scanning would be 
dynamic and interactive, with concomitant perfor- 
mance measures. 

The second major advance that has facilitated 
the study of spatial navigation is an explosion in the 
development of computer simulation technology. 
Commercially available video games are dynamic 
and interactive, with a first-person ground-level 
perspective, and have increasingly complex and 
naturalistic large-scale environments as their back- 
drops. These virtual reality (VR) games are often 
accompanied by editors, allowing researchers to 
manipulate aspects of the game to produce envi- 
ronments and scenarios suitable to address experi- 
mental questions (see chapter 17, this volume). 
The extent of immersion or presence felt in a VR 
environment, that is, the degree to which the user 
treats it as he or she does the real world and be- 
haves in a similar manner, is obviously an impor- 
tant concern. With some limitations, several studies 
have indicated good correspondence between the 
spatial knowledge of an environment acquired in 
the real world and a model of that environment in 
VR (Arthur, Hancock, & Chrysler, 1997; Regian & 
Yadrick, 1994; Ruddle, Payne, & Jones, 1997; 
Witmer et al., 1996), and VR has been used to aid 
rehabilitation in patients with memory deficits 
(Brooks & Rose, 2003) and to teach disabled chil- 
dren (Wilson, Foreman, & Tlauka, 1996). Care 
must be taken in designing realistic VR environ- 
ments as, for example, realistic landmarks improve 
navigation while abstract coloured patterns do not 
(Ruddle et al., 1997), and performance tends to 
correlate with the extent of presence felt by the sub- 
ject (Witmer & Singer, 1994). The use of VR in 
both neuropsychological and neuroimaging con- 
texts has opened up new avenues to explore how 
the brain navigates in the real world (Burgess et al., 
2002; Maguire, Burgess, & O'Keefe, 1999; Spiers & 
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Maguire, 2006). In this chapter, I describe some of 
the most recent studies that exploit this approach. I 
believe that examining brain-environment interac- 
tions in this way is already contributing to key theo- 
retical debates in the field of memory, and may in 
the future directly inform treatment and rehabilita- 
tive interventions in memory-impaired patients. 
Furthermore, this emerging knowledge may one 
day influence environmental design itself, closing 
the loop, so to speak, between brain and space. 



The Good, the Bad, and the Lost 

One of the most enduring questions about naviga- 
tion concerns why some people are better at finding 
their way than others. Why do people get lost? In 
one early neuroimaging study, the brain regions in- 
volved in active navigation were directly investi- 
gated by requiring subjects to find their way 
between locations within a complex, texture-rich 
VR town while in a positron emission tomography 
(PET) scanner (Maguire, Burgess, Donnett, Frack- 
owiak, et al., 1998). This town was created to in- 
clude many different possible routes between any 



two locations. The right parahippocampus and hip- 
pocampus were activated by successful navigation 
between locations based on the subjects' knowledge 
of the layout of the town compared to following a 
route of arrows through the town. Most interesting 
of all, subjects' accuracy of navigation was found to 
correlate significantly with activation in the right 
hippocampus (see figure 9.1). Activation of the left 
hippocampus was associated with successful navi- 
gation but did not correlate with accuracy of navi- 
gation. By contrast, medial and right inferior 
parietal activation was associated with all condi- 
tions involving active movement through the town. 
Speed of virtual movement through the town cor- 
related with activation in the caudate nucleus, 
whereas performance of novel detours was associ- 
ated with additional activations in left prefrontal 
cortex. This study highlighted the distributed net- 
work underpinning spatial navigation in (virtual) 
large-scale space (see Spiers & Maguire, 2006, for 
more on this). Furthermore, it showed that it was 
possible to identify specific functions of particular 
brain areas, allowing us to theorize about how the 
navigation system as a whole might work (see also 
Burgess et al., 2002, for a discussion on how these 
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Figure 9.1 . Activity in the hippocampus correlates with accuracy of path taken in a virtual reality town (see 
Maguire, Burgess, Donnett, Frackowiak, et al., 1998). The more direct the path (in yellow on the aerial view), 
the more active the hippocampus. See also color insert. 
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Figure 9.2. Place-responsive cells (see Ekstrom et al., 2003) from intracranial electrodes in the human brain while 
patients navigated in a VR townlike environment. These neurons were clustered in the hippocampus (H) compared 
with the amygdala (A), parahippocampal region (PR), and frontal lobes (FR). From Ekstrom et al. (2003). 
Reprinted by permission from Macmillan Publishers Ltd.: Nature, Arne D. Ekstrom et al., Cellular networks under- 
lying human spatial navigation, volume 425, issue 6954, pages 184-188, 2003. See also color insert. 



data fit with the wider navigation literature). Ac- 
cordingly, we suggested that these results were con- 
sistent with right hippocampal involvement in 
supporting a representation of locations within the 
town allowing accurate navigation, left hippocam- 
pal involvement in more general mnemonic pro- 
cesses, posterior parietal involvement in guiding 
egocentric movement through space, orienting the 
body relative to doorways, avoiding obstacles, etc., 
and involvement of the caudate in movement- 
related aspects of navigation. 

Essentially the same task was used to investigate 
navigation following either focal bilateral hippocam- 
pal damage (Spiers, Burgess, Hartley, et al, 2001) or 
unilateral anterior temporal lobectomy (Spiers, 
Burgess, Maguire, et al., 2001). Participants were 
tested on their ability to navigate accurately to 10 
locations in the VR town. The right temporal lobec- 
tomy patients were impaired compared to controls, 
taking longer routes (Spiers, Burgess, Maguire, et al., 
2001). A patient with focal bilateral hippocampal 
pathology, Jon (Vargha-Khadem et al., 1997) was 
also tested and was impaired on the navigation task 
(Spiers, Burgess, Hartley, et al., 2001). Interestingly, 
this VR town was also used as a backdrop for an 
episodic memory task (Spiers, Burgess, Hartley, 
et al., 2001; Spiers, Burgess, Maguire, et al., 2001; 
Burgess, Maguire, Spiers, & O'Keefe, 2001). Sub- 
jects followed a prescribed route through the VR 
town and along the way repeatedly met two charac- 
ters who gave them different objects in two different 



places. In contrast to spatial navigation, the overall 
performance of the left temporal lobectomy patients 
on remembering who gave them the objects and 
when was significantly worse than controls. As well 
as being impaired on the navigation task, hippocam- 
pal patient Jon was also impaired on all the episodic 
memory tests. These data seem to confirm a role for 
the right hippocampus in supporting spatial naviga- 
tion, and possibly the left hippocampus in more 
general aspects of context-specific episodic memory. 
Ekstrom et al. (2003) provided striking confir- 
matory evidence for this navigation view at the 
single-neuron level (see figure 9.2). Patients with 
pharmacologically intractable seizures had intracra- 
nial electrodes implanted in the hippocampus, 
parahippocampal, and frontal regions. Responses 
from single neurons were recorded while patients 
navigated around a small VR town. Cells were iden- 
tified that fired when a patient was in a particular 
location in the town, irrespective of his or her orien- 
tation. These neurons were mostly found in the hip- 
pocampus and may be similar to the place cells 
identified in the rat hippocampus (e.g., O'Keefe & 
Dostrovsky, 1971). By contrast, cells that responded 
to a particular view, that is, a particular shop front, 
were mostly located in the parahippocampal cortex. 
Finally, cells that responded to the goal of the pa- 
tient, such as picking up a particular passenger, 
were distributed through frontal and temporal cor- 
tices. Previous functional neuroimaging studies that 
found hippocampal activation during navigation 
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Figure 9.3. Town 1 from Hartley et al. (2003). In the wayfinding task, the current target location is indi- 
cated in the lower right corner of the VR display The map shows an example path followed by a subject 
(solid line) between the first three target locations. The corresponding ideal path is shown as a dotted 
line. Accuracy of performance was correlated with activity in the hippocampus. See also color insert. 



(e.g., Maguire, Burgess, Donnett, Frackowiak, 1998) 
are consistent with the high density of place cells 
found in the patients' hippocampi. Not only does 
the Ekstrom study confirm previous animal and hu- 
man neuroimaging work, but in one way it also ex- 
tends it. The demonstration of goal-related neurons 
highlights an important aspect of navigation that has 
received some attention in environmental psychol- 
ogy (e.g., Magliano, Cohen, Allen, & Rodrigue, 
1995), but little in human neuroscientific navigation 
studies (see Spiers & Maguire, 2006, for a recent ex- 
ample). Every journey we make has a purpose, and 
different goals may influence the cognitive processes 
engaged, and the brain systems activated. 

While measuring the firing of single neurons is 
highly desirable, such studies are very rare, ex- 
tremely difficult to execute, and by their nature are 
highly constrained in terms of the brain areas that 
can be sampled. Functional neuroimaging therefore 
offers the next best means to study the navigating 
brain. In the last several years there have been fur- 
ther improvements in the realism of VR environ- 
ments coupled with the better spatial and temporal 
resolution of functional MRI (fMRI; see also chapter 
4, this volume). Capitalizing on these develop- 
ments, a study has provided additional insights into 
the differences between good and less good naviga- 



tors. Hartley et al. (2003) were interested in con- 
trasting two different kinds of navigation we all ex- 
perience in our everyday lives. On the one hand, 
we often navigate along very familiar routes, for in- 
stance taking the same route from work to home 
(route following). By contrast, we sometimes have 
to use novel routes locating new places in a familiar 
environment (wayfinding). Prior to being scanned, 
subjects learned two distinct but similar VR towns. 
In one, subjects repeatedly followed the same route, 
while in the other they were allowed to explore 
freely. During fMRI scanning, subjects found their 
way to specified destinations in the freely explored 
town or followed the well-worn route in the other 
town. Behavioral performance was measured by 
comparing the path taken during each task with the 
ideal path, with the distance error calculated as the 
additional distance traveled by the subject. The hip- 
pocampus was more active in good navigators and 
less active in poorer navigators during wayfinding 
(see figure 9.3). Good navigators activated the head 
of the caudate while navigating along a well-learned 
route (see figure 9.4). Hartley et al. (2003) sug- 
gested that good navigators select the appropriate 
representation for the task at hand, the hippocam- 
pal representation for wayfinding, and the caudate 
representation for route following. Consistent with 
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Figure 9.4. Town 2 from Hartley et al. (2003). In the route-following task, the current target location is indi- 
cated in the lower right corner of the VR display. The map shows the well-worn route followed by a subject 
(solid line). Accuracy of performance was correlated with activity in the caudate nucleus. See also color insert. 



this, the authors also noted that in the poorest navi- 
gators, activation in the head of the caudate was 
greatest during wayfinding, suggesting the use of an 
inappropriate (route-following) representation. 

In another fMRI study, subjects navigated in a 
virtual radial eight-arm mazelike environment 
(Iaria, Petrides, Dagher, Pike, & Bohbot, 2003). It 
was found that subjects spontaneously adopted 
one of two strategies during navigation. Some sub- 
jects used the relationship between landmarks to 
guide navigation. Other subjects used a nonspatial 
strategy whereby they counted the arms of the 
maze clockwise or counterclockwise from the start 
position or a single landmark. The authors noted 
that this suggests a natural variability in the strate- 
gies adopted by humans faced with a navigation 
task. They went on to report that increased activity 
was apparent in the right hippocampus only in 
those subjects using the spatial landmark strategy. 
By contrast, the group that adopted the nonspatial 
strategy instead showed sustained activity in the 
caudate nucleus. This and the Hartley et al. (2003) 
results suggest that the hippocampus and caudate 
systems both offer a means to support navigation 
in humans. The engagement of the most appropri- 
ate system for the task at hand may be a fundamen- 
tal contributor to the success or otherwise of the 



navigation. Of course, this begs several additional 
questions: Are the systems competing or comple- 
mentary, and what factors influence their engage- 
ment in the first place? 

Voermans et al. (2004) provided some further 
insights combining VR and fMRI with a lesion ap- 
proach. Patients with preclinical Huntington's dis- 
ease (HD) are a useful model of relatively selective 
caudate dysfunction. Subjects, both healthy con- 
trols and preclinical HD patients, had to memorize 
and recognize well-defined routes through VR 
homes in a navigational memory task. A noncom- 
petitive interaction was found such that the hip- 
pocampus compensated for gradual caudate nucleus 
dysfunction with a gradual increase in activity, 
maintaining normal behavior. Although character- 
ized here as a complementary relationship, others 
have described it as competitive (Poldrak & Packard, 
2003). It is possible that parallel systems might 
have a cooperative capacity when one system is 
damaged. However, in healthy individuals, the two 
may still compete and possibly impede one an- 
other. Given that the hippocampal and caudate 
systems have very different operating mechanisms 
(Packard & McGaugh, 1996; White & McDonald, 
2002), it may be that the flexible hippocampus can 
compensate for a compromised caudate system, 
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but the more functionally constrained caudate can- 
not fulfill a flexible navigation role. 

In the absence of pathology, what might drive a 
subject to adopt a particular navigation strategy, en- 
gaging one brain system rather than another? One 
factor that is often cited anecdotally in relation to 
navigation is sex differences. Do men and women 
differ in how they navigate? In recent years VR, 
among other methods, has been recruited to pro- 
vide more controlled means of investigating this 
question. It would seem that women might be more 
reliant on landmarks within the environment when 
navigating, with men tending to focus on the Eu- 
clidean properties of the environment (Sandstrom, 
Kaufman, & Huettel, 1998). Gron, Wunderlich, 
Spitzer, Tomczak, and Riepe (2000) suggested there 
might be a brain correlate of this difference. They 
compared a group of men with a group of women 
who navigated a VR environment during fMRI. 
They reported that the men activated the hip- 
pocampus more than the women, while the women 
activated right parietal and prefrontal cortices. It 
may be that the hippocampal activation reflects the 
male use of Euclidean information, and the cortical 
activations in the women a landmark-based strat- 
egy. However, in this study, women also performed 
significantly less well than the males on the VR task. 
Given that previous fMRI studies of exclusively 
male subjects (e.g., Hartley et al, 2003) found that 
hippocampal activity is correlated with perfor- 
mance, the sex differences seen by Gron et al. might 
be explicable in terms of a more general tendency 
for individual variations in performance to be corre- 
lated with hippocampal activation. It is certainly the 
case that women and men can perform comparably 
on VR navigation tasks, particularly when one con- 
trols for factors such as familiarity with video game 
playing. However, that women consistently do less 
well than men in navigation tests (Astur, Ortiz, & 
Sutherland, 1998; Moffat, Hampson, & Hatzipan- 
telis, 1998; Sakthivel, Patterson, & Cruz-Neira, 
1999) leaves open the question of why the female 
group performed less well in Gron et al.'s study. The 
possibility remains that the performance-related ef- 
fects observed have a physiological basis that affects 
men and women differently (perhaps for evolution- 
ary reasons; Ecuyer-Dab & Robert, 2004; Saucier et 
al., 2002). Indeed, there are structural differences 
between the medial temporal lobes of men and 
women that might index such a difference (Good et 
al., 2001). 



Understanding both sex and individual differ- 
ences in navigation is important, not least in order 
to know if one's navigation capability is preordained 
or whether it can change. Recent findings from 
studying navigation experts suggest that the human 
brain's navigation system is more plastic than hith- 
erto thought. 



The Experts 

Licensed London taxi drivers are unique. They en- 
gage in years of training (on average 2-4 years) 
in order to pass the very stringent examinations 
that enable them to qualify for a license. Some 320 
routes linking over 25,000 streets in greater 
London have to be memorized and, in addition, 
thousands of famous landmarks, buildings, and 
places of interest have to be learned. Acquiring The 
Knowledge, as it is known, is a truly amazing ac- 
complishment for the 23,000 licensed taxi drivers 
currently operating in London. Licensed London 
taxi drivers represent an ideal opportunity to ex- 
amine complex and successful navigation, and a 
previous PET study found that they activated the 
hippocampus when accurately recalling complex 
routes around the city (Maguire et al., 1997). By 
studying London taxi drivers, insights might also 
be gleaned into the effects of training on the adult 
human brain. Interestingly, neurogenesis in the 
hippocampus has now been associated with spatial 
memory and learning in birds and small mammals 
(Lavenex, Steele, & Jacobs, 2000; Patel, Clayton, & 
Krebs, 1997; Shors et al, 2001), and has been 
found in adult primates (Gould, Reeves, Graziano, 
& Gross, 1999). In general, hippocampal volume 
has been shown to be related to spatial ability 
in several species of birds and small mammals 
(Lee, Miyasato, & Clayton, 1998; Sherry, Jacobs, & 
Gaulin, 1992) in terms of their ability to keep 
track of large numbers of stored food items or 
large home ranges. Furthermore, variations in hip- 
pocampal volume in birds and small mammals 
have been found to track seasonal changes in the 
need for spatial memory (Lavenex et al., 2000; 
Smulders, Sasson, & DeVoogd, 1995). Would hip- 
pocampal volume changes be apparent in humans 
who had undergone intensive navigation training? 
The structural MRI brain scans of male licensed 
London taxi drivers have been compared with that 
of age -matched non-taxi drivers (Maguire et al., 
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Figure 9.5. MRI sagittal brain sections. Yellow areas indicate where there was increased gray matter density 
in the left (LH) and right (RH) hippocampi of licensed London taxi drivers compared with non-taxi driver 
control subjects (see Maguire et al., 2000). See also color insert. 



2000). Significant differences in gray matter volume 
between the two groups were found in the hip- 
pocampus, with the posterior hippocampus being 
larger on both sides in taxi drivers (see figure 9.5). 
Interestingly, the anterior hippocampus was smaller 
in the taxi drivers. Moreover, the increase in right 
posterior hippocampus correlated positively with 
the time spent in the job, while the anterior hip- 
pocampus decreased in volume the longer the time 
taxi driving. This study provides an intriguing hint 
of experience-dependent structural plasticity in the 
human brain and further suggests an intimate link 
with navigation and the hippocampus in humans as 
well as other animals. It also suggests that while 
training may have positive effects by increasing gray 
matter volume in one area, there may be a price to 
pay for this with a gray matter decrease elsewhere. 
The possible neuropsychological sequelae (if any) 
of the anterior hippocampal gray matter decrease in 
taxi drivers are currently under investigation. 

Is job training really the key factor that is 
driving the gray matter changes in taxi drivers? We 
hypothesised that the correlation finding suggests 
that increased posterior hippocampal gray matter 
volume is acquired in response to increased taxi 
driving experience, perhaps reflecting their detailed 



spatial representation of the city However, an alter- 
nate hypothesis is that the difference in hippocam- 
pal volume is instead associated with innate 
navigational expertise, leading to an increased likeli- 
hood of becoming a taxi driver. To investigate such a 
hypothesis requires the examination of gray matter 
in non-taxi driver navigation experts. If increased 
hippocampal gray matter volume were found to be 
taxi driver-specific, this would be further evidence 
that hippocampal structure can be changed by inter- 
action with large-scale space. To investigate this pos- 
sibility, we examined the structural MRI brain scans 
of subjects who were not taxi drivers (Maguire, 
Spiers, et al, 2003). We assembled a group of sub- 
jects who were well matched on a range of pertinent 
variables, but who showed wide variation across the 
group in terms of their ability to learn to find their 
way in a VR town. Despite this wide range of navi- 
gational expertise, there was no association between 
expertise and posterior hippocampal gray matter 
volume (or, indeed, gray matter volume throughout 
the brain). This failure to find an association be- 
tween hippocampal volume and navigational ex- 
pertise thus suggests that structural differences in 
the human hippocampus are acquired in response 
to intensive environmental stimulation. 
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What about other experts, particularly those 
who also have the ability to remember vast amounts 
of information? We examined individuals renowned 
for outstanding memory feats in forums such as the 
World Memory Championships (Maguire, Valen- 
tine, et al, 2003). These individuals practice for 
hours every day to hone the strategies that allow 
them to achieve superior memory. Using neuropsy- 
chological, structural, and functional brain imaging 
measures, we found that superior memory is not 
driven by exceptional intellectual ability or struc- 
tural brain differences. Rather, we found that supe- 
rior memorizers used a spatial learning strategy (the 
method of loci; Yates, 1966) while preferentially en- 
gaging brain regions critical for memory and for 
spatial memory in particular, including the hip- 
pocampus. It is interesting to note that, although 
very proficient in the use of this route-based spatial 
mnemonic, no structural brain changes were de- 
tected in the right posterior hippocampus such as 
were found in the London taxi drivers. This may be 
because taxi drivers develop and have a need to 
store a large and complex spatial representation of 
London, while the memorizers use and reuse a 
much more constrained set of routes. Structural 
brain differences have been reported for other pro- 
fessionals compared with nonexpert control sub- 
jects, such as orchestra musicians (Sluming et al, 
2002), pianists (Schlaug, Jancke, Huang, & Stein- 
metz, 1995), and bilinguals (Mechelli et al., 2004). 
However, the cause and effect of these differences is 
unclear. What is required is a study that tracks 
brain changes and neuropsychological performance 
in the same individuals over time while they ac- 
quire a skill. A study of this kind is currently under- 
way with individuals training to be licensed London 
taxi drivers. 

Clearly, much remains to be understood about 
how training affects the brain, and many factors 
need to be considered, not just the type of repre- 
sentation being acquired but also the nature of the 
job itself. For example, in a study examining the 
temporal lobes of an airline cabin crew, Cho (2001) 
found that chronic jet lag resulted in deficits in 
spatial memory and decreased volume in the tem- 
poral lobe. Elevated basal levels of the stress hor- 
mone Cortisol were also found in the saliva of the 
flight crews and correlated with temporal lobe 
volume reduction. Cho reasoned that the stress 
associated with disruption of circadian rhythms 
and sleep patterns produced the elevated Cortisol, 



which in turn affected temporal lobe volume. Thus 
in terms of how the brain interacts with job train- 
ing and the occupational environment in general, 
the limited number of studies so far suggest that at 
the very least both cognitive and emotional factors, 
with potential positive and negative effects, need to 
be considered. 



The Structure of the Environment 

Successful navigation in virtual or real environ- 
ments may therefore depend on the kind of brain 
doing the navigating, the type of strategy it adopts, 
and the amount of navigation exposure and training 
it has undergone. In addition to these features of 
the navigator, as we might class them, several other 
factors are clearly relevant, not least of which is the 
environment itself. Environmental psychologists 
have spent decades examining how the physical 
structure of our buildings, towns, and cities influ- 
ences how we navigate within them. How regular 
or complex the layout (Evans, 1980), how inte- 
grated the environment (Peponis, Zimring, & Choi, 
1990), and the presence of salient divisions (such as 
rivers and parks; Lynch, 1960), are just some of the 
many aspects of the physical environment they 
have studied. The relative youth of neuroscientific 
investigations of real-world environments means 
that so far there is a dearth of information about 
how the environments physical structure influences 
the brain. The single most robust finding is that the 
parahippocampal gyrus seems to be particularly 
responsive to features of the environment such 
as landmarks and buildings (e.g., Epstein & Kan- 
wisher, 1998; Maguire, Burgess, Donnett, O'Keefe, & 
Frith, 1998), in contrast to the hippocampus, which 
is more concerned with representing the overall 
spatial layout (Burgess et al, 2002). 

An fMRI study gives some further insights into 
the factors affecting the parahippocampal respon- 
sivity to landmarks. Janzen and van Turennout 
(2004) had subjects watch footage of a route 
through a VR museum that they were instructed to 
learn (see figure 9.6). The museum contained land- 
marks consisting of objects on tables, positioned 
at two types of junctions, either points where a 
navigational decision had to be made or at simple 
turns where no decision was required. Volunteers 
were then scanned using fMRI during an old-new 
recognition test in which the landmarks from the 
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Figure 9.6. Views from the virtual museum from Janzen and van Turennout (2004). The left panel shows a toy 
placed at a decision point, and the right scene a toy at a nondecision point. A parahippocampal region was more 
active for toys at decision points, that is, that had navigational relevance. From Janzen and van Turennout (2004). 
Reprinted by permission from Macmillan Publishers Ltd. : Nature Neuroscience, Gabriele Janzen and Miranda van 
Turennout, Selective neural representation of objects relevant for navigation, volume 7, issue 6, pages 673-677, 
2004. See also color insert. 



museum and new landmarks were shown from a 
canonical perspective on a white background. The 
authors found that the parahippocampal gyrus was 
more active for landmarks that had been seen at de- 
cision points than those that had been seen at sim- 
ple turns. The parahippocampal signal was apparent 
even when the navigationally relevant landmarks 
were lost to conscious awareness. This study shows 
that the brain identifies landmarks at key decision 
points, and it does so automatically, requiring just 
one exposure. This mechanism, where the associa- 
tion with navigational relevance can be made de- 
spite a change in perspective, could be an important 
basis for successful and flexible wayfinding (Spiers 
& Maguire, 2004). 



Another feature of the environment that has 
been found to elicit a specific brain response is 
when suddenly a previously used route becomes 
blocked and one is required to replan and seek an 
alterative route. When this occurred in a VR town 
during PET scanning, the left prefrontal cortex was 
more active when subjects successfully took a de- 
tour and reached their destination by this alterna- 
tive route (Maguire, Burgess, Donnett, Frackowiak, 
1998; Spiers & Maguire, 2006). This ability to take 
a detour is fundamental to flexible navigation, 
and much more remains to be understood about 
how frontal executive processes interact with the 
mnemonic role of the medial temporal lobe to 
achieve this. 
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The Learning Context 

While only tentative steps have as yet been made 
into probing the brain-physical environment rela- 
tionship, there has been some interest in exam- 
ining the effect of learning context on brain 
responses. How we acquire spatial knowledge 
might influence how well we subsequently navi- 
gate. Learning to find our way in a new area is nor- 
mally accomplished by exploration at the ground 
level. Alternatively, we often use an aerial or survey 
perspective in the form of maps to aid navigation. 
Shelton and Gabrieli (2002; see also Mellet et al., 
2000) scanned subjects using fMRI while they 
learned two different VR environments, one from 
the ground level and the other from a survey 
perspective. Behaviorally in these rather simple en- 
vironments, subsequently tested performance ac- 
curacy was similar for the two types of perspective. 
While there were commonalities in some of the 
brain areas active during the two kinds of learning, 
there were also differences. Learning from the 
ground perspective activated additional areas in- 
cluding the hippocampus and the right inferior 
parietal and posterior cingulate cortices. By con- 
trast, learning from an aerial perspective activated 
fusiform and inferior temporal areas, superior pari- 
etal cortex, and the insula. The authors suggested 
that the different patterns of brain activation might 
reflect the psychological differences between 
ground and aerial encoding. Subjects reported that 
at the ground level they had a sense of immersion 
in the environment, whereas survey learners had 
no such feeling. Ground-level learning also re- 
quires much more updating of information as the 
local environment changes as one moves through it 
(see also Wolbers, Weiller, & Buchel, 2004), while 
the aerial perspective allows more direct access to 
the global environmental structure. The greater hip- 
pocampal activation in the ground-level learning 
may therefore reflect the spatial updating and map- 
building properties of this brain region (O'Keefe & 
Nadel, 1978). While the accuracy of performance 
did not differ between the two learning perspec- 
tives in this instance, in more complex and realistic 
environments differences might emerge. This study 
highlights that how an environment is initially ex- 
perienced may influence the type of spatial repre- 
sentation acquired and the kinds of purposes for 
which it might be suited. 



Virtual reality provides an opportunity to inves- 
tigate another feature of the learning context in hu- 
man spatial memory, namely viewpoint dependence. 
While we often learn a landmark or route from a 
particular point of view or direction, truly flexible 
navigation requires the ability to find our way from 
any direction. King, Burgess, Hartley, Vargha- 
Khadem, and O'Keefe (2002) provided subjects with 
a view from the rooftops surrounding a richly tex- 
tured VR courtyard. During presentation, objects ap- 
peared in different locations around the courtyard. 
During testing, several copies of each object were 
presented in different locations, with the subject 
asked to indicate which was in the same location as 
at presentation. Between presentation and testing, 
the subject's viewpoint might remain the same or be 
changed to another location overlooking the court- 
yard. A patient with focal bilateral hippocampal 
pathology, Jon (Vargha-Khadem et al., 1997) was 
mildly impaired on the same-view condition. In con- 
trast, he was massively impaired in the shifted view- 
point condition, suggesting a crucial role for the 
human hippocampus in representing a world- 
centered or allocentric view of large-scale space. A 
recent fMRI study has also documented brain activa- 
tion differences between viewer-centered, object- 
centered and landmark-centered frames of reference 
in a VR environment (Committeri et al., 2004). 



Conclusion 

Even 15 years ago, being able to make meaningful 
inferences from observing brain activity while peo- 
ple navigate around complex environments seemed 
unthinkable. This highly selective review illustrates 
that today we are able to do just that. Real-world set- 
tings are no longer out of bounds for the experimen- 
talist. Rather, I would argue they are essential to fully 
appreciate the true operation of the human brain. 
Technical advances in brain imaging hardware, in- 
creasing sophistication in fMRI experimental design 
and data analyses, as well as ever-more realistic VR 
towns and cities have all played their part in provid- 
ing this exciting opportunity. It is still early and the 
work so far has mainly focused on addressing basic 
questions and assessing convergence of evidence 
with other established fields such as animal physiol- 
ogy and human neuropsychology. However, we are 
now beginning to move on from this, appreciating 
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the plasticity and dynamics within the brain's navi- 
gation systems. It is not a stretch to hope that in the 
next few years, a much more fruitful exchange will 
be possible whereby technological and environmen- 
tal improvements might be driven by an informed 
understanding of how the brain finds its way in the 
real world. 



Main Points 

1 . It is very difficult to investigate experimentally 
the neural basis of realistic navigation in 
humans. 

2. In the last decade or so, the development of 
virtual reality and brain scanning techniques 
such as fMRI has opened up new 
opportunities. 

3. We are starting to understand the distributed 
brain networks that underpin our ability to 
navigate in large-scale space, including the 
specific contributions of regions such as the 
hippocampus, parahippocampal cortex, and 
caudate nucleus. 

4. The success or otherwise of navigation may be 
influenced by the extent to which the 
hippocampus is engaged. 

5. The type of navigation strategy employed, as 
well as factors such as gender, amount of 
navigation training, the learning context, and 
the structure of the physical environment, are 
also key factors influencing how the brain 
navigates. 
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Joel S. Warm and Raja Parasuraman 



Cerebral Hemodynamics 
and Vigilance 



The efficiency and safety of many complex human- 
machine systems can be critically dependent on 
the mental workload and vigilance of the operators 
of such systems. As pointed out by Wickens and 
Hollands (2000), it has long been recognized that 
the design of a high-quality human-machine sys- 
tem is not just a matter of assessing performance 
but also of evaluating how well operators can meet 
the workload demands imposed on them by the 
system. Major questions that must be addressed 
are whether human operators can meet additional 
unexpected demands when they are otherwise 
overloaded (Moray, 1979; Wickens, 2002) and 
whether they are able to maintain vigilance and re- 
spond effectively to critical events that occur at un- 
predictable intervals (Warm & Dember, 1998). 

These considerations point to the need for sen- 
sitive and reliable measurement of human mental 
workload and vigilance. Behavioral measures, such 
as accuracy and speed of response to probe events, 
have been widely used to assess these psychological 
functions. However, as discussed by Kramer and 
Weber (2000), Parasuraman (2003), and Wickens 
(1990), measures of brain function offer some 
unique advantages that can be exploited in particu- 
lar applications. Among these is the ability to ex- 
tract covert physiological measures continuously in 



complex system operations in which overt behav- 
ioral measures may be relatively sparse. Perhaps a 
more compelling rationale is that measures of brain 
function can be linked to emerging cognitive neuro- 
science knowledge on attention (Parasuraman & 
Caggiano, 2002; Posner, 2004), thereby allowing for 
the development of neuroergonomic theories that in 
turn can advance practical applications of research 
on mental workload and vigilance. 

In this chapter, we describe a series of recent 
neuroergonomic studies from our research group 
on vigilance, focusing on the use of noninvasive 
measurement of cerebral blood flow velocity. We 
use a theoretical framework of attentional resources 
(Kahneman, 1973; Moray, 1967; Navon & Gopher, 
1979; Norman & Bobrow, 1975; Posner & Tudela, 
1997; Wickens, 1984). Resource theory is the dom- 
inant theoretical approach to the assessment of hu- 
man mental workload (Wickens, 2002) and also 
provides a major conceptual framework for under- 
standing human vigilance performance (Parasura- 
man, 1979; Warm & Dember, 1998). Consistent 
with the view first proposed by Sir Charles Sher- 
rington (Roy & Sherrington, 1890), a considerable 
amount of research on brain imaging indicates that 
there is a close tie between cerebral blood flow and 
neural activity in the performance of mental tasks 
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(Raichle, 1998; Risberg, 1986). Consequently, 
changes in blood flow velocity and oxygenation in 
our studies are considered to reflect the availability 
and utilization of information processing assets 
needed to cope with the vigilance task. 



The Hemodynamics of Vigilance 

Brain Systems and Vigilance 

Vigilance involves the ability of observers to detect 
transient and infrequent signals over prolonged pe- 
riods of time. That aspect of human performance 
is of considerable concern to human factors and 
ergonomic specialists because of the critical role 
that vigilance occupies in many complex human- 
machine systems, including military surveillance, 
air-traffic control, cockpit monitoring and airport 
baggage inspection, industrial process and quality 
control, and medical functions such as cytological 
screening and vital sign monitoring during sur- 
gery (Hancock & Hart, 2002; Parasuraman, 1986; 
Warm & Dember, 1998). Thus, it is important to 
understand the neurophysiological factors that 
control vigilance performance. 

In recent years, brain imaging studies using 
positron emission tomography (PET) and func- 
tional magnetic resonance imaging (fMRI) tech- 
niques have been successful in demonstrating that 
changes in cerebral blood flow and glucose metab- 
olism are involved in the performance of vigilance 
tasks (see review by Parasuraman, Warm, & See, 
1998). These studies have also identified several 
brain regions that are active in such tasks, includ- 
ing the right frontal cortex and the cingulate gyrus, 
as well as subcortical nuclei such as the locus 
coeruleus. Although these studies have identified 
brain regions involved in vigilance, Parasuraman et 
al. (1998) have pointed out some major limitations 
of this research. With the exception of PET studies 
by Paus et al. (1997) and by Coull, Frackowiak, 
and Frith (1998), the brain imaging studies have 
neglected to link the systems they have identified 
to performance efficiency, perhaps because of the 
high cost associated with using PET and fMRI dur- 
ing the prolonged running times characteristic of 
vigilance research. Thus, the functional role of the 
brain systems identified in the imaging studies re- 
mains largely unknown. Gazzaniga, Ivry, and Man- 
gun (2002) have also emphasized the necessity of 



linking neuroimaging results to human performance 
for enhanced understanding of research in cogni- 
tive neuroscience. 

Other problems with the PET and fMRI proce- 
dures are that they feature restrictive environments 
in which observers need to remain almost motion- 
less throughout the scanning procedure so as not 
to compromise the quality of the brain images, and 
fMRI acquisition is accompanied by loud noise. 
Observers in vigilance experiments rarely remain 
motionless, however. Instead, research has shown 
that they tend to fidget during the performance of a 
vigilance task, with the amount of motor activity 
increasing with time on task (Galinsky, Rosa, 
Warm, & Dember, 1993). Moreover, the noise 
of fMRI is one of several environmental variables 
that can degrade vigilance performance. For exam- 
ple, Becker, Warm, Dember, and Hancock (1995) 
showed that noise lowered perceptual sensitivity 
in a vigilance task, interfered with the ability of 
observers to profit from knowledge of results, and 
elevated perceived mental workload. Accordingly, 
the conditions required for the effective use of 
the PET and fMRI techniques may not provide 
a suitable environment for linking changes in 
brain physiology with vigilance performance over a 
prolonged period of time. To meet this need, we 
turned to two other imaging procedures, transcra- 
nial Doppler sonography (TCD) and transcranial 
cerebral oximetry. 

Transcranial Doppler Sonography 

TCD is a noninvasive neuroimaging technique that 
employs ultrasound signals to monitor cerebral 
blood flow velocity or hemovelocity in the main- 
stem intracranial arteries — the middle, anterior, 
and posterior arteries. These arteries are readily 
isolated through a cranial "transtemporal window" 
and exhibit discernible measurement character- 
istics that facilitate their identification (Aaslid, 
1986). The TCD technique uses a small 2 MHz 
pulsed Doppler transducer to gauge arterial blood 
flow. The transducer is placed just above the zygo- 
matic arch along the temporal bone, a part of the 
skull that is functionally transparent to ultrasound. 
The depth of the pulse is adjusted until the desired 
intracranial artery (e.g., the middle cerebral artery, 
MCA) is isonated. TCD measures the difference in 
frequency between the outgoing and reflected en- 
ergy as it strikes moving erythrocytes. 
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The low weight and small size of the trans- 
ducer and the ability to embed it conveniently in a 
headband permit real-time measurement of cere- 
bral blood flow while not limiting, or being ham- 
pered by, body motion. Therefore, TCD enables 
inexpensive, continuous, and prolonged monitor- 
ing of cerebral blood flow velocity concurrent with 
task performance. Blood flow velocities, measured 
in centimeters per second, are typically highest in 
the MCA, and the MCA carries about 80% of the 
blood flow within each cerebral hemisphere (Toole, 
1984). Consequently, our TCD studies of mental 
workload and vigilance assess blood flow velocity 
in the MCA, but other TCD studies, particularly 
those examining perceptual processes, also measure 
blood flow in the posterior cerebral artery (PCA). 
For further methodological details of the TCD tech- 
nique, see chapter 6. 

When a particular area of the brain becomes 
metabolically active, as in the performance of men- 
tal tasks, by-products of this activity such as car- 
bon dioxide (C0 2 ) increase. This increase in C0 2 
leads to a dilation of blood vessels serving that 
area, which in turn results in blood flow to that re- 
gion (Aaslid, 1986). Consequently, TCD offers the 
possibility of measuring changes in metabolic ac- 
tivity during task performance. The use of TCD in 
brain imaging performance applications is limited, 
in part, by its low spatial resolution: TCD can sup- 
ply gross hemispheric data, but it does not provide 
information about changes in specific brain loci, as 
is the case with PET and fMRI. Nevertheless, TCD 
offers good temporal resolution (Aaslid, 1986) and, 
compared to PET and fMRI, it can track rapid 
changes in blood flow dynamics that can be fol- 
lowed in real time under less restrictive and inva- 
sive conditions. The use of TCD to index blood 
flow changes in a wide variety of cognitive, per- 
ceptual, and motor tasks has been reviewed else- 
where (Duschek & Schandry, 2003; Klingelhofer, 
Sander, & Wittich, 1999; Stroobant & Vingerhoets, 
2000; see also, chapter 6). 

Transcranial Cerebral Oximetry 

The TCD technique provides a very economical 
way to assess cerebral blood flow in relatively unre- 
stricted environments. However, TCD does not di- 
rectly provide information on oxygen utilization 
in the brain, which would be useful to assess as 



another indicator of the activation of neuronal 
populations recruited in the service of cognitive 
processes. Optical imaging, in particular near- 
infrared spectroscopy (NIRS), can be used in the 
assessment of cerebral oximetry. There are several 
types of NIRS technology, including the recent de- 
velopment of so-called fast NIRS, as discussed in 
chapter 5. The standard NIRS technique has sev- 
eral advantages over TCD, including the ability to 
assess activation in several brain regions, and not 
just in the left and right hemispheres as with TCD. 
Previous research using NIRS has shown that tissue 
oxygenation increases with the information pro- 
cessing demands of the task being performed by 
an observer (Punwani, Ordidge, Cooper, Amess, & 
Clemence, 1998; Toronov et al., 2001). Hence, one 
might expect that along with cerebral blood flow, 
cerebral oxygenation would also be related to both 
mental workload and to vigilance. 



TCD Studies of Vigilance 

Working Memory and Vigilance 

The initial study from our research group to exam- 
ine TCD in relation to vigilance was conducted by 
Mayleben (1998). That study was guided by the 
finding that working memory demand can be a po- 
tent influence on vigilance performance (Davies & 
Parasuraman, 1982; Parasuraman, 1979). Parasura- 
man (1979) had first showed that successive dis- 
crimination tasks, in which the detection of critical 
targets requires comparison of information in work- 
ing memory, are more susceptible to performance 
decrement over time than simultaneous discrimina- 
tion tasks, which have no such memory imperative. 
The role of memory representation in the vigilance 
decrement was confirmed in a study by Caggiano 
and Parasuraman (2004). Moreover, Warm and 
Dember (1998) conducted a series of studies show- 
ing that other psychophysical and task factors that 
reduce attentional resources (e.g., low signal salience, 
dual-task demands) have a greater detrimental ef- 
fect on successive than on simultaneous vigilance 
tasks. These findings can be interpreted in terms of 
the resource model described earlier in this chapter. 
According to that model, a limited-capacity infor- 
mation processing system allocates resources to 
cope with situations that confront it. The vigilance 
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Figure 10.1. Mean cerebral blood flow velocity as a function of periods of 
watch for simultaneous-type (SIM) and successive-type (SUC) vigilance tasks. 
Error bars are standard errors. After Mayleben (1998). 



decrement, the decline in signal detections over 
time that characterizes vigilance performance 
(Davies & Parasuraman, 1982; Warm & Jerison, 
1984), reflects the depletion of information pro- 
cessing resources or reservoirs of energy that cannot 
be replenished in the time available. Given that 
changes in blood flow might reflect the availability 
and utilization of the information processing assets 
needed to cope with a vigilance task, Mayleben 
(1998) hypothesized that the vigilance decrement 
should be accompanied by a decline in cerebral he- 
movelocity and that the overall level of blood flow 
should be greater for a memory-demanding succes- 
sive task than for a memory-free simultaneous task. 
Participants in this study were asked to per- 
form either a successive or a simultaneous vigi- 
lance task during a 30-minute vigil. Critical signals 
for detection in the simultaneous task were cases 
in which one of two lines on a visual display was 
slightly taller than the other. In the successive task, 
critical signals were cases in which both lines were 
slightly taller than usual. Pilot work ensured that 
the tasks were equated for difficulty under alerted 
conditions. In this and in all of the subsequent 
studies from our laboratory described in this chap- 



ter, blood flow or hemovelicity is expressed as a 
percentage of the last 60 seconds of a 5-minute 
resting baseline, as recommended by Aaslid (1986). 
As illustrated in figure 10.1, Mayleben (1998) 
found that the vigilance decrement in detection rate 
over time was accompanied by a parallel decline in 
cerebral hemovelocity. Also consistent with expec- 
tations from a resource model, the overall level of 
blood flow velocity was significantly higher for ob- 
servers who performed the successive task than for 
those who performed the simultaneous task. 

An important additional finding of the 
Mayleben (1998) study was that the blood flow ef- 
fects were lateralized — hemovelocity was greater in 
the right than in the left hemisphere, principally in 
the performance of the memory-based successive 
task. A result of this sort is consistent with earlier 
PET and psychophysical studies showing right- 
brain superiority in vigilance (Parasuraman et al., 
1998) and with studies by Tulving, Kapur, Craik, 
Moscovitch, and Houle (1994) indicating that 
memory retrieval is primarily a right-brain function. 
Schnittger, Johannes, Arnavaz, and Munte (1997) 
also reported the performance-blood flow relation 
over time described in the Mayleben (1998) study. 
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However, a clear coupling of blood flow and perfor- 
mance could not be determined in their investiga- 
tion because of the absence of a control for the 
possibility of spontaneous declines in blood flow 
over time, such as may result from systemic de- 
clines in arousal. Following a suggestion by Para- 
suraman (1984), Mayleben (1998) employed such 
a control by exposing a group of observers to the 
dual-line display for 30 minutes in the absence of a 
work imperative. Blood flow remained stable over 
the testing period under such conditions. Thus, the 
decline in cerebral blood flow was closely linked to 
the need to maintain attention to the visual display 
and not merely to the passage of time. 

A potential challenge to an interpretation of 
these results along resource theory lines comes from 
the findings that blood flow velocity is sensitive to 
changes in blood pressure and cardiac output (Ca- 
plan et al., 1990) and that changes in heart rate 
variability are correlated with vigilance performance 
(Parasuraman, 1984). Accordingly, one could argue 
that the performance and hemovelocity findings in 
this study do not reflect information processing per 
se but rather a gross change in systemic vascular ac- 
tivity that covaried with blood flow. The lateraliza- 
tion of the performance and hemovelocity findings 
challenges such a view, since gross changes in vas- 
cular activity are not likely to be hemisphere de- 
pendent. 

Controlling the Vigilance Decrement 
with Signal Cueing 

Signal detection in vigilance can be improved by 
providing observers with consistent and reliable 
cues to the imminent arrival of critical signals. As 
previous experiments have shown, the principal 
consequence of such forewarning is the elimina- 
tion of the vigilance decrement (Annett, 1996; 
Wiener & Attwood, 1968). The cueing effect can 
be linked to resource theory as follows. Observers 
need to monitor a display only after having been 
prompted about the arrival of a signal, and there- 
fore can husband their information processing re- 
sources over time. In contrast, when no cues are 
provided, observers are never certain of when a 
critical signal might appear, and consequently 
have to process information on their displays con- 
tinuously across the watch, thereby consuming 
more of their resources over time than cued ob- 
servers. Thus, one can predict that in the presence 



of perfectly reliable cueing, the temporal decline 
in cerebral blood flow would be attenuated in 
comparison to a noncued condition and also in 
comparison to conditions in which cueing was less 
than perfectly reliable, since observers in such 
conditions would not be relieved of the need to at- 
tend continuously to the vigilance display. 

This prediction was tested by Hitchcock et al. 
(2003) using a simulated air-traffic control (ATC) 
display. Critical signals for detection were pairs of 
aircraft traveling on a potential collision course. Ob- 
servers monitored the simulated ATC display for 40 
minutes. To manipulate perceptual difficulty, signal 
salience, varied by changing the Michaelson contrast 
ratio of the aircraft to their background, was high 
(98%, dark black aircraft on a light background) or 
low (2%, light gray aircraft on a light background). 
Signal salience was combined factorially with four 
levels of cue reliability — 100% reliable, 80% reli- 
able, 40% reliable, and a no-cue control. Observers 
in the cueing groups were instructed that a critical 
signal would occur within one of the five display 
updates immediately following the verbal prompt 
look provided through a digitized male voice. Ob- 
servers in each of the cue groups were advised 
about the reliability of the cues they would receive. 
To control for accessory auditory stimulation, ob- 
servers in the no-cue group received acknowledg- 
ment after each response in the form of the word 
logged spoken in the same male voice. 

As can be seen in figure 10.2, the detection 
scores for the several cueing conditions in this study 
were similar to each other during the early portion 
of the vigil and diverged by the end of the session. 
More specifically, performance efficiency remained 
stable in the 100% reliable cueing condition but de- 
clined over time in the remaining conditions, so 
that by the end of the vigil, performance efficiency 
was clearly best in the 100% group followed in or- 
der by the 80%, 40%, and no-cue groups. 

The hemovelocity scores from the left hemi- 
sphere showed a significant decline over time but 
no effect for cueing with either high-salience or 
low-salience signals. A similar result was found for 
high-salience signals in the right hemisphere. Cue- 
ing effects emerged, however, in the hemovelocity 
scores for the right hemisphere with low-salience 
signals. As was the case with detection probability, 
the hemovelocity scores for the several cueing con- 
ditions were similar to each other during the early 
portions of the vigil, but showed differential rates 
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Figure 10.2. Percentages of correct detections as 
a function of periods of watch for four cue-reliability 
conditions. After Hitchcock et al. (2003). 



of decline over time, so that by the end of the vigil, 
blood flow was clearly highest in the 100% group 
followed in order by the 80%, 40%, and no cue 
groups. This result is illustrated in figure 10.3. 

In summary, the hemovelocity scores taken 
from the right hemisphere under low salience al- 
most exactly mirrored the effects of cuing on per- 
formance efficiency. The finding that this result was 
limited to low-salience signals is consistent with 
a study by Korol and Gold (1998) indicating that 
brain systems involving glucose metabolism need 
to be sufficiently challenged in order for measur- 
able physiological changes to emerge in cognitive 
and attentional processing tasks. Restriction of 
the cue-time-salience hemovelocity findings to the 
right hemisphere is consistent with expectations 
about right hemisphere control of vigilance. As in 



the initial study, blood flow remained stable over 
time in both hemispheres throughout the watch 
when observers were exposed to the simulated air- 
traffic display without a work imperative. 

Visual Search 

The ATC task used in the Hitchcock et al. (2003) 
study was such that critical signals could be de- 
tected without any substantial need for searching 
the display. In contrast, many real-world environ- 
ments, both in ATC and elsewhere, require that op- 
erators conduct a visual search of displays in order 
to detect critical signals. A well-established finding 
from laboratory studies of visual search is the search 
asymmetry phenomenon (Treisman & Gormican, 
1988). This effect refers to more rapid detections 
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Figure 10.3. Mean hemovelocity scores as a func- 
tion of periods of watch for four cue-reliability condi- 
tions. Data are from the right hemisphere/low-signal 
salience condition. After Hitchcock et al. (2003). 
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when searching for the presence of a distinguishing 
feature in an array of stimuli as opposed to its ab- 
sence. Indeed, when searching for presence, the 
distinguishing feature appears to be so salient that it 
seems to pop out of the display. The phenomenon 
of search asymmetry has been accounted for by the 
feature integration model (Treisman & Gormican, 
1988), which suggests that searching for the pres- 
ence of a feature is guided by preattentive, parallel 
processing, while more deliberate serial processing 
is required for determining its absence. 

Studies by Schoenfeld and Scerbo (1997, 
1999) have extended the presence-absence distinc- 
tion to the accuracy of signal detections in long- 
duration sustained attention or vigilance tasks. 
Performance efficiency in vigilance tasks varies in- 
versely with the information processing demand 
imposed by the task, as indexed by the number of 
stimulus elements that must be scanned in search 
of critical signals (Grubb, Warm, Dember, & Berch, 
1995; Parasuraman, 1986). The view that detecting 
the absence of a feature is more capacity demand- 
ing than detecting its presence led Schoenfeld and 
Scerbo (1997, 1999) to predict that increments in 
the number of array elements to be scanned in sep- 
arating signals from noise in vigilance would have a 
more negative effect upon signal detection in the 
feature absence than in the feature presence case. 
Consistent with that prediction, they found that 
when observers were required to detect the ab- 
sence of a figure, signal detectability declined as 
the size of the stimulus array to be scanned was in- 
creased from two to five elements. Increasing array 
size, however, had no effect on performance when 
observers were required to monitor for the pres- 
ence of that feature. In addition, observers rated 
the workload of their assignment to be greater 
when monitoring for feature absence than presence 
on the NASA Task Load Index (TLX) scale, a stan- 
dard subjective report measure of the perceived 
mental workload imposed by a task (Hart & Stave- 
land, 1988; Warm, Dember, & Hancock, 1996; 
Wickens & Hollands, 2000). 

In most vigilance tasks, critical signals for detec- 
tion are embedded in a background of repetitive 
nonsignal or neutral events. Several studies have 
demonstrated that signal detections vary inversely 
with the background event rate and that this effect is 
more prominent in tasks requiring high information 
processing demand than low (Lanzetta, Dember, 
Warm, & Berch, 1987; Parasuraman, 1979; Warm & 



Jerison, 1984). Given that detecting the absence of a 
feature is more capacity demanding than detecting 
its presence, Hollander and his associates (2004) 
hypothesized that the degrading effects of incre- 
ments in background event rate would be more pro- 
nounced when observers monitored for the absence 
than for the presence of a feature. As in the studies 
by Schoenfeld and Scerbo (1997, 1999), perceived 
mental workload was also anticipated to be greater 
when observers monitored for feature absence than 
for feature presence. With regard to blood flow, Hol- 
lander et al. (2004) predicted that blood flow would 
be higher when observers were required to detect 
feature absence than presence and would show a 
greater decline over time in the absence than in the 
presence condition. The two types of tasks, presence 
and absence, were combined factorially with three 
levels of event rate, 6,12, and 24 events per minute, 
to provide six experimental conditions. In all condi- 
tions, observers participated in a 40-minute vigil di- 
vided into four 10-minute periods of watch during 
which they monitored an array of five circles posi- 
tioned around the center of a video display terminal 
at the 3, 5, 7, 9, and 12 o'clock locations. The criti- 
cal signal for detection in the presence condition 
was the appearance of a vertical 4 mm line intersect- 
ing the 6 o'clock position within one of the circles in 
the array. In the absence condition, the vertical line 
was present in all circles but one. Ten critical signals 
were presented in each watchkeeping period in all 
experimental conditions. 

As anticipated, the event rate effect was indeed 
more pronounced in the absence than in the pres- 
ence condition. Detection probability remained sta- 
ble across event rates in the presence condition but 
declined with increments in event rate in the ab- 
sence condition. Another important aspect of the 
performance data was the finding that signal detec- 
tions declined significantly over time in both the 
feature presence and absence conditions. 

The finding that the event rate effect was more 
pronounced when observers had to monitor for 
stimulus absence than presence is reminiscent of 
the findings in the earlier reports by Schoenfeld 
and Scerbo (1997, 1999) that the effect of another 
information processing factor in vigilance, the size 
of the element set that must be scanned in search 
of critical signals, is also more notable in the fea- 
ture absence than the feature presence condition. 
Also consistent with Schoenfeld and Scerbo (1997, 
1999), perceived mental workload was greater 
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Figure 10.4. Mean hemovelocity scores in feature presence or absence conditions for 
successive 1 -minute intervals during the initial period of watch. Data are for the right 
hemisphere. Error bars are standard errors. After Hollander et al. (2004). 



when participants monitored for feature absence 
than presence. Thus, the results support the notion 
of differential capacity demand in detecting feature 
absence than presence. However, the finding of a 
vigilance decrement in the feature presence condi- 
tion suggests that counter to the early claim in 
feature-integration theory that feature detection is 
preattentive, some information processing cost 
must be associated with detecting feature presence. 
This interpretation is supported by the fact that 
even though the mental workload of the presence 
condition was less than that of the absence condi- 
tion, it still fell in the upper level of the NASA TLX 
scale. Spatial cueing studies in which the size of a 
precue is varied prior to the presentation of the 
search display have also found cue-size effects on 
target identification time for both feature and con- 
junction search (Greenwood & Parasuraman, 1999, 
2004). Results such as these are consistent with the 
emerging view in the search literature that the 
alignment of feature detection with purely preat- 
tentive processing may no longer be tenable (Pash- 
ler, 1998; Quinlan, 2003). 

When measured over the 10-minute intervals 
of watch, blood flow in the Hollander et al. (2004) 
study was found to be greater in the presence than 
in the absence condition. That result was counter 
to expectations based on the view that detecting 
feature absence is more capacity demanding than 
detecting feature presence. It is conceivable, how- 
ever, that this apparent reversal of the expected ef- 
fect reflected the fact than the demands of feature 



absence were great enough to tax information pro- 
cessing resources very early in the vigil and that 
those resources were not replenished over time. An 
account along that line would be supported if it 
could be shown that while differences in blood 
flow are greater in the feature absence than in the 
feature presence case at the very outset of the vigil, 
the reverse effect emerged as the vigil continued. 
Toward that end, a fine-grained minute-by-minute 
analysis was performed on the blood flow scores of 
the presence and absence conditions during the 
initial watchkeeping period in the left and right 
hemispheres. No task differences were noted for the 
left hemisphere. However, as shown in figure 10.4, 
blood flow velocity in the right hemisphere was 
greater in the feature absence than in the feature 
presence condition at the outset of the vigil, and 
the reverse effect emerged after observers had per- 
formed the task for 6 minutes. Statistical tests indi- 
cated that there were no significant differences 
between the two tasks in the first 5 minutes of 
watch but that there were statistically significant 
differences between the conditions in the 6th 
through the 10th minutes of watch. Similar fine- 
grained examination of the data for the remaining 
periods of watch revealed that the reduced level of 
blood flow in the absence condition that emerged 
halfway through the initial watchkeeping period 
remained consistent throughout each minute of all 
of the remaining periods of watch. 

Evidently, Hollander et al.'s (2004) initial expec- 
tation of greater blood flow in the absence condition 
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Figure 10.5. Percentage change inhe- 
movelocity relative to resting baseline in 
the left and right cerebral hemispheres for 
the control and active vigilance condi- 
tions. After Helton et al. (in press). 
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underestimated the degree to which that condition 
taxed information processing resources in the vigi- 
lance task. Rather than being reflected in an overall 
elevation in blood flow, the greater information 
processing demand exerted by the absence con- 
dition was evident in an early-appearing drain 
on resources. Although initially unanticipated, this 
finding from the fine-grained analysis of the data is 
consistent with the expectation of a steeper decline 
in blood flow associated with the absence con- 
dition. As in the Hitchcock et al. (2003) and 
Mayleben (1998) studies, hemovelocity in the Hol- 
lander et al. (2004) study remained stable over the 
course of the watch in both hemispheres among 
control observers who viewed the displays without 
a work imperative, indicating once again that the 
blood flow effects were indeed task dependent. 

The Abbreviated Vigil 

Thus far we have discussed TCD findings in stud- 
ies that made use of traditional vigilance tasks last- 
ing 30 minutes or more. Because of their long 
duration, investigators have found it inconvenient 
to incorporate such tasks in test batteries or, as dis- 
cussed previously, to link vigilance performance 
with brain imaging metrics such as PET and fMRI. 
Accordingly, it is of interest to examine whether the 
TCD-vigilance findings can be replicated in shorter- 
duration vigilance tasks (Nuechterlein, Parasura- 
man, & Jiang, 1983; Posner, 1978; Temple et al., 
2000). Toward that end, Helton et al. (in press) 
used a 12-minute vigilance task developed by Tem- 
ple et al. (2000) in which participants were asked 



to inspect the repetitive presentation on a VDT of 
light gray capital letters consisting of an O, a D, 
and a backward D. The letters were presented for 
only 40 ms at a rate of 57.5 events per minute and 
exposed against a visual mask consisting of unfilled 
circles on a white background. Critical signals for 
detection were the appearances of the letter O (sig- 
nal probability = 0.20/period of watch). In addi- 
tion to TCD measurement of blood flow, Helton et 
al. (in press) also employed the NIRS procedure to 
measure transcranial cerebral oximetry via a Soma- 
netics INVOS 4100 Cerebral Oximeter. 

As can be seen in figures 10.5 and 10.6, both 
blood flow velocity (figure 10.5) and oxygenation 
(figure 10.6) were found to be significantly higher 
in the right than in the left cerebral hemisphere 
among observers who performed the vigilance task 
while there were no hemispheric differences in the 
blood flow and oxygenation measures among con- 
trol observers who viewed the vigilance display 
without a work imperative. In this study, the oxy- 
genation measure was based on a percentage of a 
3-minute resting baseline. 

Clearly, the results of this study indicated that 
performance in the abbreviated vigil was right lat- 
eralized, a finding that coincides with the outcome 
of earlier blood flow studies featuring more tradi- 
tional long-duration vigils and with PET and fMRI 
investigations (Parasuraman et al., 1998). This par- 
allel has several important implications. It provides 
strong support for the argument that the abbrevi- 
ated vigil is a valid analog of long-duration vigi- 
lance tasks. The fact that the NIRS procedure 
yielded laterality effects similar to those of the TDC 
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Figure 10.6. Percentage change frontal lobe 
oxygenation relative to resting baseline in 
the left and right cerebral hemispheres for 
the control and active vigilance conditions. 
After Helton et al. (in press). 



procedure further implies that laterality in vigi- 
lance is a generalized effect that appears in terms of 
both hemovelocity and blood oxygenation. It also 
implies that the NIRS procedure may be a useful 
supplement to the TCD approach in providing a 
noninvasive imaging measure of brain activity in 
the performance of a vigilance task. 

It is important to note that while Helton et al.'s 
(in press) results regarding laterality of function 
with the abbreviated vigil were consistent with 
those found with its long-duration analogs, their 
findings regarding the decrement function were 
not. Both the TCD and the NIRS indices remained 
stable over the course of the watch while perfor- 
mance efficiency declined over time. It is possible 
that cerebral hemodynamics are structured so that 
overall hemispheric dominance emerges early in the 
time course of task performance but that temporally 
based declines in cerebral blood flow and blood 
oxygen levels require a considerable amount of time 
to become observable. Thus, the abbreviated 12- 
minute vigil, which is only about 30% as long as 
the vigils employed in earlier vigilance studies, may 
not be long enough to permit time -based declines 
in blood flow or blood oxygen levels. 



Conclusion 



measures of cerebral blood flow to assess human 
mental workload and vigilance can be considered a 
success. The vigilance studies have revealed a close 
coupling between vigilance performance and blood 
flow and they provide empirical support for the no- 
tion that blood flow may represent a metabolic in- 
dex of information processing resource utilization 
during sustained attention. The demonstration of 
systematic modulation of blood flow in the right ce- 
rebral hemisphere with time on task, memory load, 
signal salience and cueing, the detection of feature 
absence or presence, and target detection in Temple 
et al.'s (2000) abbreviated vigil provides evidence 
for a right hemispheric brain system that is involved 
in the functional control of vigilance performance 
over time. 

Another goal of neuroergonomics research is to 
use knowledge of brain function to enhance human- 
system performance. In additional to the theoretical 
and empirical contributions of TCD research, there 
are also some potentially important ergonomic ram- 
ifications. TCD may offer a noninvasive and inex- 
pensive tool to "monitor the monitor" and to help 
decide when operator vigilance has reached a point 
where task aiding is necessary or operators need to 
be rested or removed. NIRS-based measurement of 
blood oxygenation may provide similar information. 



One of the goals of neuroergonomics is to enhance 
understanding of aspects of human performance in 
complex systems with respect to the underlying 
brain mechanisms and to provide measurement 
tools to study these mechanisms (Parasuraman, 
2003). From this perspective, the use of TCD-based 



Main Points 

1. Transcranial Doppler sonography and near- 
infrared spectroscopy can be used to measure 
cerebral blood flow velocity and cerebral 
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oxygenation, respectively, during the perfor- 
mance of a vigilance task. 

2. The temporal decline in signal detections that 
characterizes vigilance performance is 
accompanied by a similar decline in brain 
blood flow over time. 

3. Brain blood flow is greater in the performance 
of a memory-demanding successive-type 
vigilance task than for a memory-free 
simultaneous-type task. 

4. Changes in signal detection in vigilance 
brought about by variations in cue reliability 
are paralleled by changes in brain blood flow 

5. The temporal decline in blood flow 
accompanying vigilance performance is greater 
when critical signals for detection are defined 
by the absence than by the presence of a target 
element. 

6. Both the TCD and NIRS measures point to a 
right hemispheric system in the control of 
vigilance performance. 
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Jordan Grafman 



Executive Functions 



There is no region of the human cerebral cortex 
whose functional assignments are as puzzling to us 
as the human prefrontal cortex (HPFC). Over 100 
years of observation and experimentation has led to 
several general conclusions about its overall func- 
tions. The prefrontal cortex is important for modu- 
lating higher cognitive processes such as social 
behavior, reasoning, planning, working memory, 
thought, concept formation, inhibition, attention, 
and abstraction. Each of these processes is very im- 
portant for many aspects of human ergonomic 
study. Yet, unlike the research conducted in other 
cognitive domains such as object recognition or 
word storage, there has been little effort to propose, 
and investigate in detail, the underlying cognitive 
architecture that would capture the essential fea- 
tures and computational properties of the higher 
cognitive processes presumably modulated by the 
HPFC. Since the processes that are attributed to the 
HPFC appear to constitute the most complex and 
abstract of human cognitive functions, many of 
which are responsible for the internal guidance of 
behavior, a critical step in understanding the func- 
tions of the human brain requires an adequate de- 
scription of the cognitive topography of the HPFC. 
In this chapter, I argue for the validity of a 
representational research framework to understand 



HPFC functioning in humans. My colleagues and I 
have labeled the set of HPFC representational units 
as a structured event complex (SEC). I briefly sum- 
marize the key elements of the biology and struc- 
ture of the HPFC, the evidence of its importance 
in higher-level cognition based on convergent evi- 
dence from lesion and neuroimaging studies, and 
some key models postulating the functions of the 
HPFC, and finally offer some suggestions about how 
HPFC functions are relevant for ergonomic study. 



Anatomical Organization of 
the Human Prefrontal Cortex 

What we know about the anatomy and physiology 
of the HPFC is inferred almost entirely from work 
in the primate and lower species. It is likely that the 
connectivity already described in other species also 
exists in the HPFC (Petrides & Pandya, 1994). The 
HPFC is composed of Brodmann's areas 8-14 and 
24-47. Grossly, it can be subdivided into lateral, 
medial, and orbital regions with Brodmann's areas 
providing morphological subdivisions within (and 
occasionally across) each of the gross regions (Bar- 
bas, 2000). Some regions of the prefrontal cortex 
have a total of six layers; other regions are agranular, 



159 



160 



Perception, Cognition, and Emotion 



meaning that the granule cell layer is absent. The 
HPFC has a columnar design like other cortical re- 
gions. All regions of the HPFC are interconnected. 
The HPFC is also richly interconnected with other 
areas of brain and has at least five distinct regions 
that are independently involved in separate corti- 
costriatal loops (Alexander, Crutcher, & DeLong, 
1990). The functional role of each relatively segre- 
gated circuit has been described (Masterman & 
Cummings, 1997). The HPFC also has strong lim- 
bic system connections via its medial and orbital ef- 
ferent connections that terminate in the amygdala, 
thalamus, and parahippocampal regions (Groe- 
newegen & Uylings, 2000; Price, 1999). Finally, the 
HPFC has long pathway connections to association 
cortex in the temporal, parietal, and occipital lobes. 
Almost all of these pathways are reciprocal. 

When compared with the prefrontal cortex 
of other species, most investigators have claimed 
that the HPFC is proportionally (compared to the 
remainder of the cerebral cortex) much larger 
(Rilling & Insel, 1999; Semendeferi, Armstrong, 
Schleicher, Zilles, & Van Hoeseall, 2001). Other re- 
cent research indicates that the size of the HPFC is 
not proportionally larger than that of other pri- 
mates, but that its internal neural architecture must 
be more sophisticated, or at least differentially or- 
ganized in order to support superior human func- 
tions (Chiavaras, LeGoualher, Evans, & Petrides, 
2001; Petrides & Pandya, 1999). The functional 
argument is that in order to subserve such higher- 
order cognitive functions as extended reactive plan- 
ning and complex reasoning that are not obviously 
apparent in other primates or lower species, the 
HPFC must have a uniquely evolved neural archi- 
tecture (Elston, 2000). 

The HPFC is not considered fully developed 
until the mid to late 20s. This is later than almost 
all other cortical association areas. The fact that the 
HPFC does not fully mature until young adulthood 
suggests that those higher cognitive processes me- 
diated by the prefrontal cortex are still developing 
until that time (Diamond, 2000). 

The HPFC is innervated by a number of differ- 
ent neurotransmitter and peptide systems — most 
prominent among them being the dopaminergic, 
serotonergic, and cholinergic transmitters and their 
varied receptor subtypes (Robbins, 2000). The 
functional role of each of these neurotransmitters 
in the HPFC is not entirely clear. Mood disorders 
that involve alterations in serotonergic functions 



lead to reduced blood flow in HPFC. Several de- 
generative neurological disorders are at least par- 
tially due to disruption in the production and 
transfer of dopamine from basal ganglia structures 
to the HPFC. This loss of dopamine may cause 
deficits in cognitive flexibility. Serotonergic recep- 
tors are distributed throughout the HPFC and have 
a role in motivation and intention. Finally, the 
basal forebrain in ventral and posterior HPFC is 
part of the cholinergic system, whose loss can 
cause impaired memory and attention. These mod- 
ulating chemical anatomical systems may be im- 
portant for adjusting the "gain" within and across 
representational networks in order to facilitate or 
inhibit activated cognitive processes. 

A unique and key property of neurons in the 
prefrontal cortex of monkeys (and presumably 
humans) is their ability to fire during an interval 
between a stimulus and a delayed probe (Levy & 
Goldman-Rakic, 2000). Neurons in other brain ar- 
eas are either directly linked to the presentation of 
a single stimulus or the probe itself, and if they 
demonstrate continuous firing, it is probable that 
they are driven by neurons in the prefrontal cortex 
or by continuous environmental input. If the firing 
of neurons in the prefrontal cortex is linked to ac- 
tivity that moves the subject toward a goal rather 
than reacting to the appearance of a single stimu- 
lus, then potentially those neurons could continu- 
ously fire across many stimuli or events until the 
goal was achieved or the behavior of the subject 
disrupted. This observation of sustained firing of 
prefrontal cortex neurons across time and events 
has led many investigators to suggest that the HPFC 
must be involved in the maintenance of a stimulus 
across time, that is, working memory (Fuster, Bod- 
ner, & Kroger, 2000). 

Besides the property of sustained firing, Elston 
(2000) has demonstrated a unique structural fea- 
ture of neurons in the prefrontal cortex. Elston 
(2000) found that pyramidal cells in the prefrontal 
cortex of macaque monkeys are significantly more 
spinous than pyramidal cells in other cortical areas, 
suggesting that they are capable of handling a 
larger amount of excitatory input than pyramidal 
cells elsewhere. This could be one of several struc- 
tural explanations for the HPFCs ability to integrate 
input from many sources in order to implement 
more abstract behaviors. 

Thus, the HPFC is a proportionally large cor- 
tical region that is extensively and reciprocally 
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interconnected with other associative, limbic, and 
basal ganglia brain structures. It matures somewhat 
later than other cortex, is richly innervated with 
modulatory chemical systems, and may have some 
unique structural features not found in other corti- 
cal networks. Finally, neurons in the prefrontal cor- 
tex appear to be particularly able to fire over 
extended periods of time until a goal is achieved. 
These features of the HPFC map nicely onto some 
of the cognitive attributes of the HPFC identified in 
neuropsychological and neuroimaging studies. 



Functional Studies of Human 
Prefrontal Cortex 

The traditional approach to understanding the 
functions of the HPFC is to perform cognitive stud- 
ies testing the ability of normal and impaired hu- 
mans on tasks designed to induce the activation of 
processes or representational knowledge presum- 
ably stored in the HPFC (Grafman, 1999). Both an- 
imals and humans with brain lesions can be studied 
to determine the effects of a prefrontal cortex lesion 
on task performance. Lesions in humans, of course, 
are due to an act of nature, whereas lesions in ani- 
mals are precisely and purposefully made. Likewise, 
intact animals can be studied using precise electro- 
physiological recordings of single neurons or neural 
assemblies. In humans, powerful new neuroimaging 
techniques such as functional magnetic resonance 
imaging (fMRI) have been used to demonstrate 
frontal lobe activation during the performance of a 
range of tasks in normal subjects and patients (see 
also chapter 4). A potential advantage in studyin 
humans (instead of animals) comes from the pre 
sumption that since the HPFC represents the kind 
of higher-order cognitive processes that distinguish 
humans from other primates, an understanding of 
its underlying cognitive and neural architecture 
can only come from the study of humans. 

Patients with frontal lobe lesions are generally 
able to understand conversation and commands, 
recognize and use objects, express themselves ade- 
quately to navigate through some social situations in 
the world, learn and remember routes, and even 
make decisions. On the other hand, they have 
documented deficits in sustaining their attention 
and anticipating what will happen next, in dividing 
their resources, inhibiting prepotent behavior, 
adjusting to some situations requiring social cogni- 



tion, processing the theme or moral of a story, form- 
ing concepts, abstracting, reasoning, and planning 
(Arnett et al., 1994; Carlin et al., 2000; Dimitrov, 
Grafman, Soares, & Clark, 1999; Dimitrov, Granetz, 
et al, 1999; Goel & Grafman, 1995; Goel et al., 
1997; Grafman, 1999; Jurado, Junque, Vendrell, 
Treserras, & Grafman, 1998; Vendrell et al, 1995; 
Zahn, Grafman, & Tranel, 1999). These deficits 
have been observed and confirmed by investigators 
over the last 50 years of clinical and experimental 
research. 

Neuroimaging investigators have published 
studies that show prefrontal cortex activation 
during encoding, retrieval, decision making and 
response conflict, task switching, reasoning, plan- 
ning, forming concepts, understanding the moral 
or theme of a story, inferring the motives or inten- 
tions of others, and similar high-level cognitive 
processing (Goel, Grafman, Sadato, & Hallett, 
1995; Koechlin, Basso, Pietrini, Panzer, & Graf- 
man, 1999; Koechlin, Corrado, Pietrini, & Graf- 
man, 2000; Nichelli et al., 1994; Nichelli, Grafman, 
et al., 1995; Wharton et al., 2000). The major ad- 
vantage, so far, of these functional neuroimaging 
studies is that they have generally provided conver- 
gent evidence for the involvement of the HPFC in 
controlling endogenous and exogenous-sensitive 
cognitive processes, especially those that are en- 
gaged by the abstract characteristics of a task. 



Neuropsychological Frameworks 
to Account for HPFC Functions 



§ Working Memory 

Working memory has been described as the cogni- 
tive process that allows for the temporary activa- 
tion of information in memory for rapid retrieval 
or manipulation (Ruchkin et al., 1997). It was first 
proposed some 30 years ago to account for a vari- 
ety of human memory data that were not ad- 
dressed by contemporary models of short-term 
memory (Baddeley 1998b). Of note is that subse- 
quent researchers have been unusually successful 
in describing the circumstances under which the 
so-called slave systems employed by working 
memory would be used. These slave systems al- 
lowed for the maintenance of the stimuli in a num- 
ber of different forms that could be manipulated by 
the central executive component of the working 
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memory system (Baddeley 1998a). Neuroscience 
support for their model followed quickly. Joaquin 
Fuster was among the first neuroscientists to recog- 
nize that neurons in the prefrontal cortex appeared 
to have a special capacity to discharge over time in- 
tervals when the stimulus was not being shown 
prior to a memory-driven response by the animal 
(Fuster et al., 2000). He interpreted this neuronal 
activity as being concerned with the cross-temporal 
linkage of information processed at different points 
in an ongoing temporal sequence. Goldman-Rakic 
and her colleagues later elaborated on this notion 
and suggested that these same PFC neurons were 
fulfilling the neuronal responsibility for working 
memory (Levy & Goldman-Rakic, 2000). In her 
view, PFC neurons temporarily hold in active 
memory modality-specific information until a re- 
sponse is made. This implies a restriction on the 
kind of memory that may be stored in prefrontal 
cortex. That is, this point of view suggests that 
there are no long-term representations in the pre- 
frontal cortex until an explicit intention to act is re- 
quired, and then a temporary representation is 
created. Miller has challenged some of Goldman- 
Rakic's views about the role of neurons in the pre- 
frontal cortex and argued that many neurons in the 
monkey prefrontal cortex are modality nonspecific 
and may serve a broader integrative function rather 
than a simple maintenance function (Miller, 2000). 
Fuster, Goldman-Rakic, and Baddeley's programs 
of research have had a major influence on the func- 
tional neuroimaging research programs of Court- 
ney (Courtney, Petit, Haxby, & Ungerleider, 1998), 
Smith and Jonides (1999), and Cohen (Nystrom 
et al., 2000) — all of whom have studied normal 
subjects in order to remap the HPFC in the context 
of working memory theory. 

Executive Function and 
Attentional/Control Processes 

Although rather poorly described in the cognitive 
science literature, it is premature to simply dismiss 
the general notion of a central executive (Baddeley, 
1998a; Grafman & Litvan, 1999b). Several investi- 
gators have described the prefrontal cortex as the 
seat of attentional and inhibitory processes that 
govern the focus of our behaviors and therefore, 
why not ascribe the notion of a central execu- 
tive operating within the confines of the HPFC? 



Norman and Shallice (1986) proposed a dichoto- 
mous function of the central executive in HPFC. 
They argued that the HPFC was primarily special- 
ized for the supervision of attention toward un- 
expected occurrences. Besides this supervisory 
attention system, they also hypothesized the exis- 
tence of a contention scheduling system that was 
specialized for the initiation and efficient running 
of automatized behaviors such as repetitive rou- 
tines, procedures, and skills. Shallice, Burgess, 
Stuss, and others have attempted to expand this 
idea of the prefrontal cortex as a voluntary control 
device and have further fractionated the supervi- 
sory attention system into a set of parallel attention 
processes that work together to manage complex 
multitask behaviors (Burgess, 2000; Burgess, Veitch, 
de Lacy Costello, & Shallice, 2000; Shallice & 
Burgess, 1996; Stuss et al., 1999). 

Social Cognition and Somatic Marking 

The role of the HPFC in working memory and exec- 
utive processes has been extensively examined, but 
there is also substantial evidence that the prefrontal 
cortex is involved in controlling certain aspects of 
social and emotional behavior (Dimitrov, Graf- 
man, & Hollnagel, 1996; Dimitrov, Phipps, Zahn, & 
Grafman, 1999). Although the classic story of the 
19th-century patient Phineas Gage, who suffered a 
penetrating prefrontal cortex lesion, has been used 
to exemplify the problems that patients with ven- 
tromedial prefrontal cortex lesions have in obeying 
social rules, recognizing social cues, and making 
appropriate social decisions, the details of this so- 
cial cognitive impairment have occasionally been 
inferred or even embellished to suit the enthusiasm 
of the storyteller — at least regarding Gage (Macmil- 
lan, 2000). On the other hand, Damasio and his 
colleagues have consistently confirmed the associa- 
tion of ventromedial prefrontal cortex lesions and 
social behavior and decision-making abnormalities 
(Anderson, Bechara, Damasio, Tranel, & Damasio, 
1999; Bechara, Damasio, & Damasio, 2000; Bechara, 
Damasio, Damasio, & Lee, 1999; Damasio, 1996; 
Eslinger, 1998; Kawasaki et al., 2001). The exact 
functional assignment of that area of HPFC is still 
subject to dispute, but convincing evidence has 
been presented that indicates it serves to associate 
somatic markers (autonomic nervous system mod- 
ulators that bias activation and decision making) 
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with social knowledge, enabling rapid social deci- 
sion making — particularly for overlearned associa- 
tive knowledge. The somatic markers themselves 
are distributed across a large system of brain re- 
gions, including limbic system structures such as 
the amygdala (Damasio, 1996). 

Action Models 

The HPFC is sometimes thought of as a cognitive 
extension of the functional specialization of the 
motor areas of the frontal lobes (Gomez Beldarrain, 
Grafman, Pascual-Leone, & Garcia-Monco, 1999) 
leading to the idea that it must play an essential 
cognitive role in determining action sequences in 
the real world. In keeping with that view, a number 
of investigators have focused their investigations 
on concrete action series that have proved difficult 
for patients with HPFC lesions to adequately per- 
form. By analyzing the pattern of errors committed 
by these patients, it is possible to construct cogni- 
tive models of action execution and the role of the 
HPFC in such performance. In some patients, while 
the total number of errors they commit is greater 
than that seen in controls, the pattern of errors com- 
mitted by patients is similar to that seen in controls 
(Schwartz et al., 1999). Reduced arousal or effort can 
also contribute to a breakdown in action production 
in patients (Schwartz et al, 1999). However, other 
studies indicate that action production impairment 
can be due to a breakdown in access to a semantic 
network that represents aspects of action schema 
and prepotent responses (Forde & Humphreys, 
2000). Action production must rely upon an associ- 
ation between the target object or abstract goal and 
specific motoric actions (Humphreys & Riddoch, 
2000). In addition, the magnitude of inhibition of 
inappropriate actions appears related to the strength 
in associative memory of object-goal associations 
(Humphreys & Riddoch, 2000). Retrieving or rec- 
ognizing appropriate actions may even help subjects 
subsequently detect a target (Humphreys & Rid- 
doch, 2001). It should be noted that action disorga- 
nization syndromes in patients are usually elicited 
with tasks that have been traditionally part of the 
examination of ideomotor or ideational praxis, such 
as brushing your teeth, and it is not clear whether 
findings in patients performing such tasks apply to 
a breakdown in action organization at a higher level 
such as planning a vacation. 



Computational Frameworks 

A number of computational models of potential 
HPFC processes as well as of the general architec- 
ture of the HPFC have been developed in recent 
years. Some models have offered a single explana- 
tion for performance on a wide range of tasks. For 
example, Kimberg and Farah (1993) showed that 
the weakening of associations within a working 
memory component of their model led to impaired 
simulated performance on a range of tasks such as 
the Wisconsin Card Sorting Test and the Stroop 
Test that patients with HPFC lesions are known to 
perform poorly on. In contrast, other investigators 
have argued for a hierarchical approach to model- 
ing HPFC functions that incorporates a number 
of layers, with the lowest levels regulated by the 
environment and the highest levels regulated by 
internalized rules and plans (Changeux & De- 
haene, 1998). In addition to the cognitive levels of 
their model, Changeux and Dehaene, relying on 
simulations, suggested that control for transient 
"prerepresentations" that are modulated by reward 
and punishment signals improved their model's 
ability to predict patient performance data on the 
Tower of London test. Norman and Shallice (1986) 
first ascribed two major control systems to the 
HPFC. As noted earlier in this chapter, one system 
was concerned with rigid, procedurally based, and 
overlearned behaviors, whereas the other system 
was concerned with supervisory control over novel 
situations. Both systems could be simultaneously 
active, although one system's activation usually 
dominated performance. The Norman and Shallice 
model has been incorporated into a hybrid compu- 
tational model that blends their control system 
idea with a detailed description of selected action 
sequences and their errors (Cooper & Shallice, 
2000). The Cooper and Shallice model can ac- 
count for sequences of response, unlike some re- 
current network models, and like the Changeux 
and Dehaene model is hierarchical in nature and 
based on interactive activation principles. It also 
was uncanny in predicting the kinds of errors of 
action disorganization described by Schwartz and 
Humphreys in their patients. Other authors have 
implemented interactive control models that use 
production rules with scheduling strategies for ac- 
tivation and execution to simulate executive con- 
trol (Meyer & Kieras, 1997). Tackling the issue of 
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how the HPFC mediates schema processing, 
Botvinick and Plaut (2000) have argued that schemas 
are emergent system properties rather than explicit 
representations. They developed a multilayered re- 
current connectionist network model to simulate 
action sequences that is somewhat similar to the 
Cooper and Shallice model described above. In 
their simulation, action errors occurred when noise 
in the system caused an internal representation for 
one scenario to resemble a pattern usually associ- 
ated with another scenario. Their model also indi- 
cated that noise introduced in the middle of a 
sequence of actions was more disabling than noise 
presented closer to the end of the task. 

The biological plausibility of all these models 
has not been formally compared yet but it is just as 
important to determine whether these models can 
simulate the behaviors and deficits of interest. The 
fact that models such as the ones described above 
are now being implemented is a major advance in 
the study of the functions of the HPFC. 



Commonalities and Weaknesses 
of the Frameworks Used 
to Describe HPFC Functions 

The cognitive and computational models briefly 
described above have commonalities that point to 
the general role of the prefrontal cortex in main- 
taining information across time intervals and inter- 
vening tasks, in modulating social behavior, in the 
integration of information across time, and in the 
control of behavior via temporary memory repre- 
sentations and thought rather than allowing behav- 
ior to depend upon environmental contingencies 
alone. None of the major models have articulated 
in detail the domains and features of a representa- 
tional knowledge base that would support such 
HPFC functions, making these models difficult to 
reject using error or response time analysis of pa- 
tient data or functional neuroimaging. 

Say I was to describe cognitive processing in 
the cerebral cortex in the following way. The role of 
the cortex is to rapidly process information and en- 
code its features, and to bind these features to- 
gether. This role is rather dependent on bottom-up 
environmental input but represents the elements 
of this processed information in memory. Perhaps 
this is not too controversial a way to describe the 
role of the occipital, parietal, or temporal cortex in 



processing objects or words. For the cognitive neu- 
ropsychologist, however, it would be critical to 
define the features of the word or object, the char- 
acteristics of the memory representation that lead 
to easier encoding or retrieval of the object or 
word, and the psychological structure of the repre- 
sentational neighborhood (how different words or 
objects are related to each other in psychological 
and potentially neural space). Although there are 
important philosophical, psychological, and bio- 
logical arguments about the best way to describe 
a stored unit of memory (be it an orthographic 
representation of a word, a visual scene, or a con- 
ceptualization), there is general agreement that 
memories are representations. There is less agree- 
ment as to the difference between a representation 
and a cognitive process. It could be argued that 
processes are simply the sustained temporary acti- 
vation of one or more representations. 

My view is that the descriptions of the func- 
tional roles of the HPFC summarized in most of 
the models and frameworks already described in 
this chapter are inadequate to obtain a clear under- 
standing of its role in behavior. To obtain a clear 
understanding of the HPFC, I believe that a theory 
or model must describe the cognitive nature of 
the representational networks that are stored in the 
prefrontal cortex, the principles by which the 
representations are stored, the levels and forms of 
the representations, hemispheric differences in the 
representational component stored based on the 
underlying computational constraints imposed by 
the right and left prefrontal cortex, and it must lead 
to predictions about the ease of retrieving represen- 
tations stored in the prefrontal cortex under normal 
conditions, when normal subjects divide their cog- 
nitive resources or shift between tasks, and after var- 
ious forms of brain injury. None of the models noted 
above were intended to provide answers to any of 
these questions except in the most general manner. 



Process Versus Representation — How 
to Think About Memory in the HPFC 

My framework for understanding the nature of the 
knowledge stored in the HPFC depends upon the 
idea that unique forms of knowledge are stored in 
the HPFC as representations. In this sense, a repre- 
sentation is an element of knowledge that, when ac- 
tivated, corresponds to a unique brain state signified 
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by the strength and pattern of neural activity in a 
local brain sector. This representational element is 
a "permanent" unit of memory that can be strength- 
ened by repeated exposure to the same or a similar 
knowledge element and is a member of a local psy- 
chological and neural network composed of multi- 
ple similar representations. Defining the specific 
forms of the representations in HPFC so that a cog- 
nitive framework can be tested is crucial since an 
inappropriate representational depiction can com- 
promise a model or theory as a description of a tar- 
geted phenomenon. It is likely that these HPFC 
representations are parsed at multiple grain sizes 
(that are shaped by behavioral, environmental, and 
neural constraints). 

What should a representational theory claim? 
It should claim that a process is a representation 
(or set of representations) in action, essentially a 
representation that, when activated, stays activated 
over a limited or extended time domain. In order 
to be activated, a representation has to be primed 
by input from a representation located outside its 
region or by associated representations within its 
region. This can occur via bottom-up or top-down 
information transfer. A representation, when acti- 
vated, may or may not fit within the typical time 
window described as working memory. When it 
does, we are conscious of the representation. When 
it does not, we can still process that representation, 
but we may not have direct conscious access to all 
of its contents. 

The idea that representations are embedded in 
computations performed by local neural networks 
and are permanently stored within those networks 
so that they can be easily resurrected in a similar 
form whenever that network is stimulated by the 
external world's example of that representation or 
via associated knowledge is not novel nor free of 
controversy. But similar ideas of representation 
have dominated the scientific understanding of 
face, word, and object recognition and have been 
recognized as an acceptable way to describe how 
the surface and lexical features of information 
could be encoded and stored in the human brain. 
Despite the adoption of this notion of representa- 
tion to the development of cognitive architectures 
for various stimuli based on "lower-level" stimulus 
features, the application of similar representational 
theory to better understand the functions of the 
HPFC has moved much more slowly and in a more 
limited way. 



Evolution of Cognitive Abilities 

There is both folk wisdom about, and research 
support for, the idea that certain cognitive abilities 
are uniquely captured in the human brain, with lit- 
tle evidence for these same sophisticated cognitive 
abilities found in other primates. Some examples of 
these cognitive processes include complex lan- 
guage abilities, social inferential abilities, or rea- 
soning. It is not that these and other complex 
abilities are not present in other species but proba- 
bly that they exist only in a more rudimentary 
form. 

The HPFC, as generally viewed, is most devel- 
oped in humans. Therefore, it is likely that it has 
supported the transition of certain cognitive abili- 
ties from a rudimentary level to a more sophisti- 
cated one. I have already touched upon what kinds 
of abilities are governed by the HPFC. It is likely, 
however, that such abilities depend upon a set of 
fundamental computational processes unique to 
humans that support distinctive representational 
forms in the prefrontal cortex (Grafman, 1995). My 
goal in the remainder of my chapter is to suggest 
the principles by which such unique representa- 
tions would be distinctively stored in the HPFC. 



The Structured Event Complex 

The Archetype SEC 

There must be a few fundamental principles gov- 
erning evolutionary cognitive advances from other 
primates to humans. A key principle must be the 
ability of neurons to sustain their firing and code 
the temporal and sequential properties of ongoing 
events in the environment or the mind over longer 
and longer periods of time. This sustained firing 
has enabled the human brain to code, store, and 
retrieve the more abstract features of behaviors 
whose goal or end stage would not occur until well 
after the period of time that exceeds the limits of 
consciousness in the present. Gradually in evolu- 
tion, this period of time must have extended itself 
to encompass and encode all sorts of complex 
behaviors (Nichelli, Clark, Hollnagel, & Grafman, 
1995; Rueckert & Grafman, 1996, 1998). Many 
aspects of such complex behaviors must be trans- 
lated into compressed (and multiple modes of) 
representations (such as a verbal listing of a series 
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of things to do and the same set of actions in visual 
memory) while others may have real-time rep- 
resentational unpacking (unpacking means the 
amount of time and resources required to activate 
an entire representation and sustain it for behav- 
ioral purposes over the length of time it would take 
to actually perform the activity — for example, an 
activity composed of several linked events that take 
10 minutes to perform would activate some com- 
ponent representations of that activity that would 
be active for the entire 10 minutes). 

The Event Sequence 

Neurons and assemblies firing over extended peri- 
ods of time in the HPFC process sets of input that 
can be defined as events. Along with the extended 
firing of neurons that allows the processing of be- 
haviors across time, there must have also devel- 
oped special neural parsers that enabled the editing 
of these behaviors into linked sequential but indi- 
vidual events (much the way speech can be parsed 
into phonological units or sentences into grammat- 
ical constituents) (Sirigu et al, 1996, 1998). The 
event sequences, in order to be goal oriented and 
cohere, must obey a logical sequential structure 
within the constraints of the physical world, the 
culture that the individual belongs to, and the indi- 
vidual's personal preferences. These event se- 
quences, as a whole, can be conceptualized as units 
of memory within domains of knowledge (e.g., a 
social attitude, a script that describes cooking a 
dinner, or a story that has a logical plot). We pur- 
posely labeled the archetype event sequence the 
SEC in order to emphasize that we believed it to be 
the general form of representation within the 
HPFC and to avoid being too closely tied to a par- 
ticular description of higher-level cognitive pro- 
cesses contained in story, narrative processing, 
script, or schema frameworks. 

Goal Orientation 

Structured event complexes are not random chains 
of behavior performed by normally functioning 
adults. They tend to have boundaries that signal 
their onset and offset. These boundaries can be de- 
termined by temporal cues, cognitive cues, or envi- 
ronmental or perceptual cues. Each SEC, however, 
has some kind of goal whose achievement precedes 
the offset of the SEC. The nature of the goal can be 



as different as putting a bookshelf together or 
choosing a present to impress your child on her 
birthday. Some events must be more central or im- 
portant to an SEC than others. Subjects can have 
some agreement on which ones they are when ex- 
plicitly asked. Some SECs are well structured, with 
all the cognitive and behavioral rules available for 
the sequence of events to occur, and there is a clear 
definable goal. Other SECs are ill-structured, re- 
quiring the subject to adapt to unpredictable 
events using analogical reasoning or similarity 
judgment to determine the sequence of actions on- 
line (by retrieving a similar SEC from memory) as 
well as developing a quickly fashioned goal. Not 
only are SEC goals central to its execution, but the 
process of reaching the goal can be rewarding. Goal 
achievement itself is probably routinely accompa- 
nied by a reward that is mediated by the brain's 
neurochemical systems. Depending on the salience 
of this reward cue, it can become essential to the 
subject's subsequent competent execution of that 
same or similar SEC. Goal attainment is usually ob- 
vious, and subjects can consciously move onto an- 
other SEC in its aftermath. 



Representational Format of the SEC 

I hypothesize that SECs are composed of a set of 
differentiated representational forms that would be 
stored in different regions of the HPFC but are acti- 
vated in parallel to reproduce all the SEC elements 
of a typical episode. These distinctive memories 
would represent thematic knowledge, morals, ab- 
stractions, concepts, social rules, features of specific 
events, and grammars for the variety of SECs em- 
bodied in actions, stories and narratives, scripts, 
and schemas. 

Memory Characteristics 

As just described, SECs are essentially distributed 
memory units with different components of the 
SEC stored in various regions within the prefrontal 
cortex. The easiest assumption to make, then, is 
that they obey the same principles as other mem- 
ory units in the brain. These principles revolve 
around frequency of activation based on use or 
exposure, association with other memory units, 
category specificity of the memory unit, plasticity 
of the representation, priming mechanisms, and 
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binding of the memory unit and its neighborhood 
memory units to memory units in more distant 
representational networks both in, and remote from, 
the territory of the prefrontal cortex. 

Frequency of Use and Exposure 

As a characteristic that predicts a subject's ability to 
retrieve a memory, frequency is a powerful vari- 
able. For the SEC, the higher the frequency of the 
memory units composing the SEC components, 
the more resilient they should be in the face of pre- 
frontal cortex damage. That is, it is predicted that 
patients with frontal lobe damage would be most 
preserved performing or recognizing those SECs 
that they usually do as a daily routine and most im- 
paired when asked to produce or recognize novel 
or rarely executed SECs. This retrieval deficit 
would be affected by the frequency of the specific 
kind of SEC component memory units stored in 
the damaged prefrontal cortex region. 

Associative Properties Within an HPFC 
Functional Region 

In order to hypothesize the associative properties 
of an SEC, it is necessary to adapt some general in- 
formation processing constraints imposed by each 
of the hemispheres (Beeman, 1998; Nichelli, Graf- 
man, et al., 1995; Partiot, Grafman, Sadato, Flit- 
man, & Wild, 1996). A number of theorists have 
suggested that hemispheric asymmetry of informa- 
tion coding revolves around two distinct notions. 
The left hemisphere is specialized for finely tuned 
rapid encoding that is best at processing within- 
event information and coding for the boundaries 
between events. For example, the left prefrontal 
cortex might be able to best process the primary 
meaning of an event. The right hemisphere is 
thought to be specialized for coarse slower coding, 
allowing for the processing of information that is 
more distantly related (to the information currently 
being processed) and could be adept at integrating 
or synthesizing information across events in time. 
For example, the right prefrontal cortex might be 
best able to process and integrate information 
across events in order to obtain the theme or moral 
of a story that is being processed for the first time. 
When left hemisphere fine-coding mechanisms are 
relied upon, a local memory element would be rap- 
idly activated along with a few related neighbors 



with a relatively rapid deactivation. When right 
hemisphere coarse coding mechanisms are relied 
upon, there should be weaker activation of local 
memory elements but a greater spread of activation 
across a larger neighborhood of representations 
and for a sustained period of time — even corre- 
sponding to the true duration of the SEC currently 
being processed. This dual form of coding proba- 
bly occurs in parallel with subjects shifting be- 
tween the two depending on task and strategic 
demands. Furthermore, the organization of a pop- 
ulation of SEC components within a functionally 
defined region, regardless of coding mechanisms, 
should be based on the same principles argued for 
other forms of associative representation with both 
inhibition of unrelated memory units and facilita- 
tion of neighboring (and presumably related) mem- 
ory units following activation. 

Order of Events 

The HPFC is specialized for the processing of 
events over time. One aspect of the SEC that is key 
to its representation is event order. Order is coded 
by the sequence of events. The stream of action 
must be parsed as each event begins and ends in 
order to explicitly recognize the nature, duration, 
and number of events that compose the event se- 
quence (Hanson & Hanson, 1996; Zacks & Tver- 
sky, 2001). I hypothesize that in childhood, 
because of the neural constraints of an immature 
HPFC, individual events are initially represented as 
independent memory units and only later in devel- 
opment are they linked together to form an SEC. 
Thus, in adults, there should be some redundancy 
of representation of the independent event (formed 
in childhood) and the membership of that same 
event within the SEC. Adult patients with HPFC le- 
sions would be expected to commit errors of order 
in developing or executing SECs but could wind 
up defaulting to retrieving the independently stored 
events in an attempt to slavishly carry out frag- 
ments of an activity. Subjects are aware of the se- 
quence of events that make up an SEC and can 
even judge their relative importance or centrality to 
the overall SEC theme or goal. Each event has a 
typical duration and an expected onset and offset 
time within the time frame of the entire SEC that is 
coded. The order of the independent events that 
make up a particular SEC must be routinely ad- 
hered to by the performing subject in order to 
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develop a more deeply stored SEC representation 
and to improve the subject's ability to predict the 
sequence of events. The repeated performance of 
an SEC leads to the systematic and rigidly ordered 
execution of events — an observation compatible 
with the AI notion of total order planning. In con- 
trast, new SECs are constantly being encoded, 
given the variable and occasionally unpredictable 
nature of strategic thought or environmental de- 
mands. This kind of adaptive planning in AI is 
known as partial-order planning, since event se- 
quences are composed online, with the new SEC 
consisting of previously experienced events now 
interdigitating with novel events. Since there must 
be multiple SECs that are activated in a typical day, 
it is likely that they too (like the events within an 
SEC) can be activated in sequence, or additionally 
in a cascading or parallel manner (to manage two or 
more tasks at the same time). 

Category Specificity 

There is compelling evidence that the HPFC can be 
divided into regions that have predominant con- 
nectivity with specific cortical and subcortical 
brain sectors. This has led to the hypothesis that 
SECs may be stored in the HPFC on a category- 
specific basis. For example, it appears that patients 
with ventral or medial prefrontal cortex lesions 
are especially impaired in performing social and 
reward-related behaviors, whereas patients with le- 
sions to the dorsolateral prefrontal cortex appear 
most impaired on mechanistic planning tasks 
(Dimitrov, Phipps, et al., 1999; Grafman et al., 
1996; Partiot, Grafman, Sadato, Wachs, & Hallett, 
1995; Pietrini, Guazzelli, Basso, Jaffe, & Grafman, 
2000; Zalla et al., 2000). Further delineation of 
category specificity within the HPFC awaits more 
precise testing using various SEC categories as 
stimuli (Crozier et al, 1999; Sirigu et al., 1998). 

Neuroplasticity of HPFC 

We know relatively little about the neurobiological 
rules governing plasticity of the HPFC. It is proba- 
ble that the same plasticity mechanisms that accom- 
pany learning and recovery of function in other 
cortical areas operate in the frontal lobes too (Graf- 
man & Litvan, 1999a; see also chapter 22, this vol- 
ume, for related discussion of neuroplasticity). For 



example, a change in prefrontal cortex regional 
functional map size with learning has been noted. 
Shrinkage of map size is usually associated with 
learning of a specific element of many within a cate- 
gory of representation, whereas an increase in map 
size over time may reflect the general category of 
representational form being activated (but not a 
specific element of memory within the category). 
After left brain damage, right homologous HPFC 
assumption of at least some of the functions previ- 
ously associated with Broca's area can occur. How 
the unique characteristics of prefrontal cortex neu- 
rons (e.g., sustained reentrant firing patterns or 
idiosyncratic neural architectures) interact with the 
general principles of cortical plasticity has been lit- 
tle explored to date. In terms of the flexibility of 
representations in the prefrontal cortex, it appears 
that this area of cortex can rapidly reorganize itself 
to respond to new environmental contingencies or 
rules. Thus, although the general underlying princi- 
ples of how information is represented may be sim- 
ilar within and across species, individual experience 
manifested by species or individuals within a 
species will be influential in what is stored in pre- 
frontal cortex and important to control for when in- 
terpreting the results of experiments trying to infer 
HPFC functional organization. 

Priming 

At least two kinds of priming (Schacter & Buck- 
ner, 1998) should occur when an SEC is activated. 
First of all, within an SEC, there should be prim- 
ing of forthcoming adjacent and distant events 
by previously occurring events. Thus, in the case 
of the event that indicates you are going into a 
restaurant, subsequent events such as paying the 
bill or ordering from the menu may be primed at 
that moment. This priming would activate those 
event representations even though they had not 
occurred yet. The activation might be too far be- 
low threshold for conscious recognition that the 
event has been activated, but there is a probably a 
relationship between the intensity of the primed 
activation of a subsequent event and the temporal 
and cognitive distance the current event is from 
the primed event. The closer the primed event is 
in sequence and time to the priming event, the 
more activated it should be. The second kind of 
priming induced by SEC activation would involve 
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SECs in the immediate neighborhood of the one 
currently activated. Closely related SECs (or com- 
ponents of SECs) in the immediate neighborhood 
should be activated to a lesser degree than the 
targeted SEC regardless of hemisphere. More dis- 
tantly related SECs (or components of SECs) 
would be inhibited in the dominant hemisphere. 
More distantly related SECs (or components of 
SECs) would be weakly activated, rather than in- 
hibited, in the nondominant hemisphere. 

Binding 

Another form of priming, based on the principle of 
binding (Engel & Singer, 2001) of distinct repre- 
sentational forms across cortical regions, should 
occur with the activation of an SEC. The sort of 
representational forms I hypothesize are stored in 
the HPFC, such as thematic knowledge, should be 
linked to more primitive representational forms 
such as objects, faces, words, stereotyped phrases, 
scenes, and emotions. This linkage or binding en- 
ables humans to form a distributed episodic mem- 
ory for later retrieval. The binding also enables 
priming across representational forms to occur. For 
example, by activating an event within an SEC that 
is concerned with working in the office, activation 
thresholds should be decreased for recognizing 
and thinking about objects normally found in an 
office, such as a telephone. In addition, the prim- 
ing of forthcoming events within an SEC referred 
to above would also result in the priming of the ob- 
jects associated with the subsequent event. Each 
additional representational form linked to the SEC 
should improve the salience of the bound configu- 
ration of representations. Absence of highly SEC- 
salient environmental stimuli or thought processes 
would tend to diminish the overall activation of the 
SEC-bound configuration of representations and 
bias which specific subset of prefrontal cortex repre- 
sentational components are activated. 

Hierarchical Representation of SECs 

I have previously argued for a hierarchy of SEC 
representation (Grafman, 1995). That is, I predicted 
that SECs, within a domain, would range from spe- 
cific episodes to generalized events. For example, 
you could have an SEC representing the actions and 
themes of a single evening at a specific restaurant, 



an SEC representing the actions and themes of how 
to behave at restaurants in general, and an SEC rep- 
resenting actions and themes related to eating that 
are context independent. In this view, SEC episodes 
are formed first during development of the HPFC, 
followed by more general SECs, and then the 
context-free and abstract SECs. As the HPFC ma- 
tures, it is the more general, context-free, and ab- 
stract SECs that allow for adaptive and flexible 
planning. Since these SECs do not represent specific 
episodes, they can be retrieved and applied to novel 
situations for which a specific SEC does not exist. 



Relationship to Other Forms 
of Representation 

Basal Ganglia Functions 

The basal ganglia receive direct connections from 
different regions of the HPFC, and some of these 
connections may carry cognitive commands. The 
basal ganglia, in turn, send back to the prefrontal 
cortex, via the thalamus, signals that reflect their 
own processing. Even if the basal ganglia work in 
concert with the prefrontal cortex, their exact role 
in cognitive processing is still debatable. They ap- 
pear to play a role in the storage of visuomotor se- 
quences (Pascual-Leone et al, 1993; Pascual-Leone, 
Grafman, & Hallett, 1995), in reward-related be- 
havior (Zalla et al., 2000), and in automatic cogni- 
tive processing such as overlearned word retrieval. 
It is likely that the SECs in the prefrontal cortex 
bind with the visuomotor representations stored in 
the basal ganglia to produce an integrated set of 
cognitive and visuomotor actions (Koechlin et al, 
2000; Koechlin et al., 2002; Pascual-Leone, Wasser- 
mann, Grafman, & Hallett, 1996) relevant to par- 
ticular situations. 

Hippocampus and Amygdala Functions 

Both the amygdala and the hippocampus have re- 
ciprocal connections with the prefrontal cortex. 
The amygdala, in particular, has extensive connec- 
tions with the ventromedial prefrontal cortex 
(Price, 1999; Zalla et al., 2000). The amygdala's 
signals may provide a somatic marker or cue to the 
stored representational ensemble in the ventrome- 
dial prefrontal cortex representing social attitudes, 
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rules, and knowledge. The more salient the input 
provided by the somatic cue, the more important 
the somatic marker becomes for biasing the activa- 
tion of social knowledge and actions. 

The connections between the prefrontal cortex 
and the hippocampus serve to enlist the SEC as a 
contextual cue that forms part of an episodic en- 
semble of information (Thierry, Gioanni, Degene- 
tais, & Glowinski, 2000). The more salient the 
context, the more important it becomes for en- 
hancing the retrieval or recognition of episodic 
memories. Thus, the hippocampus also serves to 
help bind the activation of objects, words, faces, 
scenes, procedures, and other information stored 
in posterior cortices and basal structures to SEC- 
based contextual information such as themes or 
plans. Furthermore, the hippocampus may be in- 
volved in the linkage of sequentially occurring 
events. The ability to explicitly predict a subse- 
quent event requires conscious recollection of 
forthcoming events, which should require the par- 
ticipation of a normally functioning hippocampus. 
Since the hippocampus is not needed for certain 
aspects of lexical or object priming, for example, it 
is likely that components of the SEC that can also 
be primed (see above) do not require the participa- 
tion of the hippocampus. Thus subjects with am- 
nesia might gain confidence and comfort in 
interactions in a context if they were reexposed to 
the same context (SEC) that they had experienced 
before. In that case, the representation of that SEC 
would be strengthened even without later con- 
scious recollection of experiencing it. Thus, SEC 
representational priming in amnesia should be gov- 
erned by the same restraints that affect word or ob- 
ject priming in amnesia. 

Temporal-Parietal Cortex Functions 

The computational processes representing the ma- 
jor components of what we recognize as a word, 
object, face, or scene are stored in the posterior 
cortex. These representations are crucial compo- 
nents of a context and can provide the key cue to 
initiate the activation of an SEC event or its inhibi- 
tion. Thus, the linkage between anterior and poste- 
rior cortices is very important for providing 
evidence that contributes to identifying the tempo- 
ral and physical boundaries delimiting the inde- 
pendent events that make up an SEC. 



Evidence For and Against 
the SEC Framework 

The advantage of the SEC formulation of the rep- 
resentations stored in the HPFC is that it resem- 
bles other cognitive architecture models that are 
constructed so as to provide testable hypotheses re- 
garding their validity. When hypotheses are sup- 
ported, they lend confidence to the structure of the 
model as predicated by its architects. When hy- 
potheses are rejected, they occasionally lead to the 
rejection of the entire model but may also lead to a 
revised view of a component of the model. 

The other major driving forces in conceptualiz- 
ing the role of the prefrontal cortex have, in general, 
avoided the level of detail required of a cognitive 
or computational model and instead have opted 
for functional attributions that can hardly be dis- 
proved. This is not entirely the fault of the investi- 
gator as the forms of knowledge or processes stored 
in the prefrontal cortex have perplexed and eluded 
investigators for more than a century. What I have 
tried to do by formulating the SEC framework is to 
take the trends in cognitive capabilities observed 
across evolution and development, including greater 
temporal and sequential processing and more ca- 
pacity for abstraction, and assume what representa- 
tional states those trends would lead to. 

The current evidence for an SEC-type repre- 
sentational network is supportive but still rather 
sparse. SECs appear to be selectively processed by 
anterior prefrontal cortex regions (Koechlin et al., 
1999, 2000). Errors in event sequencing can occur 
with preservation of aspects of event knowledge 
(Sirigu, Zalla, Pillon, Grafman, Agid, et al, 1995). 
Thematic knowledge can be impaired even though 
event knowledge is preserved (Zalla et al., 2002). 
Frequency of the SEC can affect the ease of re- 
trieval of SEC knowledge (Sirigu, Zalla, Pillon, 
Grafman, Agid, et al., 1995; Sirigu, Zalla, Pillon, 
Grafman, Dubois, et al., 1995). There is evidence 
for category specificity in that the ventromedial 
prefrontal cortex appears to be specialized for so- 
cial knowledge processing (Dimitrov, Phipps, et al., 
1999). The HPFC is a member of many extended 
brain circuits. There is evidence that the hip- 
pocampus and the HPFC cooperate when the se- 
quence of events have to be anticipated (Dreher 
et al, 2006). The amygdala and the HPFC cooper- 
ate when SECs are goal and reward oriented or 
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emotionally relevant (Zalla et al., 2000). The basal 
ganglia, cerebellum, and HPFC cooperate as well 
(Grafman et al., 1992; Hallett & Grafman, 1997; 
Pascual-Leone et al, 1993) in the transfer of per- 
formance responsibilities between cognitive and vi- 
suomotor representations. When the SEC is novel 
or multitasking is involved, the anterior frontopo- 
lar prefrontal cortex is recruited, but when SECs 
are overlearned, slightly more posterior frontome- 
dial prefrontal cortex is recruited (Koechlin et al., 
2000). When subjects rely upon the visuomotor 
components of a task, the basal ganglia and cere- 
bellum are more involved but when subjects have 
to rely upon the cognitive aspects of the task, the 
HPFC is more involved in performance (Koechlin 
et al., 2002). Thus, there is positive evidence for 
the representation of several different SEC compo- 
nents within the HPFC. There has been little in the 
way of negative studies of this framework, but 
many predictions of the SEC framework in the ar- 
eas of goal orientation, neuroplasticity priming, as- 
sociative properties, and binding have not been 
fully explored to date and could eventually be falsi- 
fied. For the purposes of understanding the role of 
the prefrontal cortex in ergonomic understanding 
and decision making and learning, researchers 
should focus on the functions and representations 
of the HPFC as detailed above. 



Future Directions for the SEC Model 

The representational model of the structured event 
complex described above lends itself to the genera- 
tion of testable predictions or hypotheses. To reit- 
erate, like the majority of representational formats 
hypothesized for object, face, action, and word 
stores, the SEC subcomponents can each be char- 
acterized by the following features: frequency of 
exposure or activation, imaginableness, association 
to other items or exemplars in that particular rep- 
resentational store, centrality of the feature to the 
SEC (i.e., what proportional relevance the feature 
has to recognizing or executing the SEC), length of 
the SEC in terms of number of events and duration 
of each event and the SEC as a whole, implicit or 
explicit activation, and association to other repre- 
sentational forms that are stored in other areas of 
the HPFC or in more posterior cortex or subcorti- 
cal regions. 



All these features can be characterized psycho- 
metrically by quantitative values based on norma- 
tive studies using experimental methods that have 
obtained similar values for words, photographs, 
objects, and faces. Unfortunately, there have been 
only a few attempts to collect some of this data for 
SECs such as scripts, plans, and similar stimuli. If 
these values for all of the features of interest of 
an SEC were obtained, one could then make pre- 
dictions about changes in SEC performance after 
HPFC lesions. For example, one hypothesis from 
the SEC representational model described above is 
that the frequency of activation of a particular repre- 
sentation will determine its accessibility following 
HPFC lesions. Patients with HPFC lesions will have 
had many different experiences eating dinner, in- 
cluding eating food with their hands as a child, later 
eating more properly at the dining room table, eat- 
ing at fast-food restaurants, eating at favorite regular 
restaurants, and eventually eating occasionally at 
special restaurants or a brand-new restaurant. After 
an HPFC lesion of moderate size, a patient should 
be limited in retrieving various subcomponents of 
the SEC stored in the lesioned sector of the HPFC. 
Thus, such a patient would be expected to behave 
more predictably and reliably when eating dinner at 
home than when eating in a familiar restaurant, and 
worst of all when eating in a new restaurant with an 
unusual seating or dining procedure for the first 
time. The kinds of errors that would characterize the 
inappropriate behavior would depend on the partic- 
ular subcomponents of the SEC (and thus regions 
within or across hemispheres) that were damaged. 
For example, if the lesion were in the right dorsolat- 
eral prefrontal cortex, the patient might have diffi- 
culty integrating knowledge across dining events so 
that he or she would be impaired in determining the 
(unstated) theme of the dinner or restaurant, partic- 
ularly if the restaurant procedures were unfamiliar 
enough that the patient could not retrieve an analo- 
gous SEC. Only one study has attempted to directly 
test this general idea of frequency sensitivity with 
modest success (Sirigu, Zalla, Pillon, Grafman, Agid, 
et al, 1995a). This is just one example of many pre- 
dictions that emerge from the SEC model with com- 
ponents that have representational features. The 
claim that SEC representational knowledge is stored 
in the HPFC in various cognitive subcomponents is 
compatible with claims made for models for other 
forms of representational knowledge stored in other 
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areas of the brain and leads to the same kind of gen- 
eral predictions regarding SEC component accessi- 
bility made for these other forms of knowledge 
following brain damage. Future studies need to test 
these predictions. 



Representation Versus 
Process Revisited 

The kind of representational model I have pro- 
posed for the SEC balances the overreliance upon 
so-called process models such as working memory 
that dominate the field today. Process models rely 
upon a description of performance (holding or ma- 
nipulating information) without necessarily being 
concerned about the details of the form of repre- 
sentation (i.e., memory) activated that is responsi- 
ble for the performance. 

Promoting a strong claim that the prefrontal 
cortex is concerned with processes rather than 
permanent representations is a fundamental shift 
of thinking away from how we have previously 
tried to understand the format in which informa- 
tion is stored in memory. It suggests that the pre- 
frontal cortex has little neural commitment to 
long-term storage of knowledge, in contrast to the 
posterior cortex. Such a fundamental shift in brain 
functions devoted to memory requires a much 
stronger philosophical, neuropsychological, and 
neuroantomical defense for the process approach 
than has been previously offered by its propo- 
nents. The representational point of view that I 
offer regarding HPFC knowledge stores is more 
consistent with previous cognitive neuroscience 
approaches to understanding how other forms of 
knowledge such as words or objects are represented 
in the brain. It also allows for many hypotheses to 
be derived for further study and therefore can mo- 
tivate more competing representational models of 
HPFC functions. 



Neuroergonomic Applications 

There is no doubt that the impairments caused by 
lesions to the HPFC can be very detrimental to 
people's ability to maintain their previous level of 
work, responsibility to their family, and social com- 
mitments (Grafman & Litvan, 1999b). These are 
all key ergonomic and social issues. In turn, the 



general role of the prefrontal cortex in maintaining 
information across time intervals and intervening 
tasks, in modulating social behavior, in the integra- 
tion of information across time, and in the control 
of behavior via temporary memory representations 
and thought rather than allowing behavior to de- 
pend upon environmental contingencies alone ap- 
pears critical for high-level ergonomic functioning. 
We know that deficits in executive functions can 
have a more profound effect on daily activities and 
routines than sensory deficits, aphasia, or agnosia 
(Schwab et al., 1993). Rehabilitation specialists 
are aware of the seriousness of deficits in executive 
impairments, but there are precious few group 
studies detailing specific or general improvements 
in executive functions that are maintained in the 
real world and that lead to a positive functional 
outcome (Levine et al, 2000; Stablum et al, 
2000). Likewise, performance in various work sit- 
uations that rely upon HPFC cognitive processes 
have been rarely studied by cognitive neuroscien- 
tists. This is one reason why the development of 
neuroergonomics is an encouraging sign of fur- 
ther interaction between cognitive neuroscience 
and human factors (Parasuraman, 2003). Hope- 
fully, a deeper understanding of HPFC functions 
will lead to even more applications. I will briefly 
describe three examples of how executive func- 
tions, associated with the HPFC, are used in day- 
to-day life. 

The first example involves driving. Driving is 
often conceived of as requiring skills that engage 
perceptual, tactical, and strategic processes. Both 
tactical and strategic processes would require the 
development and execution of stored plans that 
favor long-term success (e.g., no tickets, no acci- 
dents) over short-term gain (e.g., expression of 
anger at other drivers by cutting them off, driving 
fast to make an appointment, looking at a map 
while driving instead of pulling off to the side of 
the road). Complicating matters these days is the 
use of cell phones that require divided attention. 
Dividing attention almost always results in a 
decrement in performance in one of the tasks be- 
ing performed, and this no doubt takes place with 
driving skills when a driver is carrying on a cell 
phone conversation while driving. So even though 
we can use a skill associated with the frontal lobes 
like multitasking that results in an increase in the 
quantity of tasks simultaneously performed, it does 
not mean the quality of your task performance will 
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improve — in fact, it is likely to decline. Over- 
learned strategies, rapid tactical decision making, 
and dividing attention are all abilities primarily 
governed by the HPFC and are likely to be needed 
in situations ranging from driving to managerial 
decisions, warfare, and air-traffic control towers. 

A second example of how executive functions 
are used in daily life involves making judgments of 
others' behaviors. The ventromedial prefrontal cor- 
tex might be concerned with storing attitudes and 
stereotypes about others (e.g., this would include 
rapid judgments about an unknown person's abili- 
ties, depending on their sex, age, ethnicity, and 
racial identity). Your ability to determine the inten- 
tion of others might depend on your ability to ac- 
cess your stored knowledge of plans and other 
behavioral sequences I referred to in this chapter as 
SECs, with the most frequent and typical SECs be- 
ing stored in the medial prefrontal cortex. Your ac- 
quired knowledge about the moral behaviors of 
known individuals would also be stored in the an- 
terior HPFC. Thus, many regions of our HPFC that 
give us a sense of inner control, wisdom, insight, 
sensitivity, and hunches are involved in judging a 
person under circumstances ranging from meeting 
a stranger to choosing a political party or candidate 
to vote for, solidifying a work relationship, or get- 
ting married. 

The third example involves stock investment. 
Making the assumption that the investor is either a 
professional or an informed day trader, many abili- 
ties associated with the human prefrontal cortex 
are involved in the needed judgments and perfor- 
mance. These abilities include modulating tenden- 
cies for short-term investments governed by great 
fluctuations in worth in favor of more secure long- 
term investments, having a rational overall plan for 
investments given an individual's needs, and plan- 
ning how much time to devote to researching dif- 
ferent kinds of investments. All of these abilities 
would depend heavily upon the HPFC. 

It is also apparent that the HPFC functions op- 
timally between the late 20s and mid-50s, and so 
expert decision making with optimal high-level 
skills should occur primarily between those ages. 
Both younger and older individuals are more at 
risk for performing at a disadvantage as high-level 
cognitive skills are demanded. In addition, there 
are wide individual differences in the ability to uti- 
lize high-level cognitive skills governed by the 
HPFC even within the optimal age range. Although 



academic ability can be assessed through tradi- 
tional tests and measures of achievement like 
graduating with a higher degree or proven perfor- 
mance skills, the higher-level cognitive abilities 
associated with the HPFC are rarely directly as- 
sessed. It might be that a new discipline of neuroer- 
gonomics would supplement traditional human 
factors research for specific tasks (such as driving, 
economic decision making, and social dynamics) 
if the addition of cognitive neuroscience tech- 
niques would improve both assessment and im- 
plementation of skills. An obvious model for this 
application would be the introduction of neuroer- 
gonomics to the training practices of government 
agencies concerned with improving the skills and 
abilities of soldiers. 



Conclusion 

In this chapter, I have shown that key higher-level 
cognitive functions known as the executive func- 
tions are strongly associated with the HPFC. I have 
argued that an important way to understand the 
functions of the HPFC is to adapt the representa- 
tional model that has been the predominant ap- 
proach to understanding the neuropsychological 
aspects of, for example, language processing and 
object recognition. The representational approach 
I developed is based on the structured event com- 
plex framework. This framework claims that there 
are multiple subcomponents of higher-level knowl- 
edge that are stored throughout the HPFC as 
distinctive domains of memory. I also have argued 
that there are topographical distinctions in where 
these different aspects of knowledge are stored in 
the HPFC. Each memory domain component of the 
SEC can be characterized by psychological features 
such as frequency of exposure, category specificity, 
associative properties, sequential dependencies, 
and goal orientation, which governs the ease of re- 
trieving an SEC. In addition, when these memory 
representations become activated via environmen- 
tal stimuli or by automatic or reflective thought, 
they are activated for longer periods of time than 
knowledge stored in other areas of the brain, giving 
rise to the impression that performance dependent 
upon SEC activation is based on a specific form of 
memory called working memory. Adapting a repre- 
sentational framework such as the SEC framework 
should lead to a richer corpus of predictions about 
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subject performance that can be rejected or vali- 
dated via experimental studies. Furthermore, the 
SEC framework lends itself quite easily to rehabili- 
tation practice. Regarding issues of importance in 
ergonomics, the HPFC is important for managing 
aspects of decision making, social cognition, plan- 
ning, foresight, goal achievement, and risk 
evaluation — all the kinds of cognitive processes 
that contribute to work-related decision making 
and judgment. Whether the application of cogni- 
tive neuroscience techniques substantially increase 
the success of training and evaluation methods 
currently adapted by the human factors commu- 
nity remains to be seen. 



Main Points 

1 . The human prefrontal cortex is the last brain 
area to develop in humans and is one of the 
few areas of the brain most evolved in 
humans. 

2. Executive functions including reasoning, 
planning, and social cognition are primarily 
mediated by the prefrontal cortex. 

3. There is a scientific debate regarding whether 
the prefrontal cortex primarily is a processor 
of knowledge stored in other brain areas or 
has its own unique knowledge stores. 

4. Executive functions (and therefore the 
prefrontal cortex) mediate many of the skills 
and abilities that are considered essential for 
high-level work performance. 
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Antoine Bechara 



The Neurology of Emotions 
and Feelings, and Their Role 
in Behavioral Decisions 



The 1970s and 1980s were replete with studies by 
decision-making researchers identifying phenom- 
ena that systematically violated normative princi- 
ples of economic behavior (Kahneman & Tversky, 
1979). Decision-making research in the 1990s be- 
gan to see a shift in emphasis from not merely 
demonstrating violations of normative principles to 
attempting to shed light on the underlying psycho- 
logical mechanisms responsible for the various ef- 
fects. Today, several researchers agree that the next 
phase of exciting research in this area is likely to 
emerge from building on recent advances in the 
field of neuroscience. 

Modern economic theory assumes that human 
decision making involves rational Bayesian maxi- 
mization of expected utility, as if humans were 
equipped with unlimited knowledge, time, and 
information-processing power. The influence of 
emotions on decision making has been ignored for 
the most part. However, the development of what 
became known as the expected utility theory was re- 
ally based on the idea that people established their 
values for wealth on the basis of the pain and plea- 
sure that it would give them. So utility was con- 
ceived as a balance between pleasure and pain. 
These notions about emotions in human deci- 
sions were eliminated from notions of utility in 



subsequent economic models. To the extent that 
current economic models of expected utility ex- 
clude emotion from their vocabulary, it is really in- 
consistent with their foundations. 

Thus the prevalent assumption, which is per- 
haps erroneous, is that a direct link exists between 
knowledge and the implementation of behavioral 
decisions, that is, one does what one actually 
knows. This is problematic in view of the fact that 
normal people often deviate from rational choice, 
despite having the relevant knowledge. This devia- 
tion is even more pronounced in patients with cer- 
tain neurological or psychiatric disorders, who 
often engage in behaviors that could harm them, 
despite clear knowledge of the consequences. 
Therefore, the neuroscience of decision making in 
general, and understanding the processes by which 
emotions exert an influence on decision making in 
particular, provides a neural road map for the in- 
tervening physiological processes between knowl- 
edge and behavior, and the potential interruptions 
that lead to a disconnection between what one 
knows and what one does not. Many of these inter- 
vening steps involve hidden physiological pro- 
cesses, many of which are emotional in nature, and 
neuroscience can enrich our understanding of a 
variety of decision-making phenomena. Given the 
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importance of emotion in the understanding of hu- 
man suffering, its value in the management of dis- 
ease, its role in social interactions, and its relevance 
to fundamental neuroscience and cognitive sci- 
ence, a comprehensive understanding of human 
cognition requires far greater knowledge of the 
neurobiology of emotion. The aim of this chapter is 
to provide a neuroscientific perspective in support 
of the view that the process of decision making is 
influenced in many important ways by neural sub- 
strates that regulate homeostasis, emotion, and 
feeling. Implications are also drawn for decision 
making in everyday and work situations. 



The Neurology of Emotions 
and Feelings 

Suppose you see a person you love bringing you a 
red rose. The encounter may cause your heart to 
race, your skin to flush, and facial muscles to con- 
tract with a happy facial expression. The encounter 
may also be accompanied by some bodily sensa- 
tions, such as hearing your heartbeat, feeling but- 
terflies in your stomach. However, there is also 
another kind of sensation, the emotional feelings 
of love, ecstasy, and elation directed toward your 
loved one. Neuroscientists and philosophers have 
debated whether these two sensations are fun- 
damentally the same. The psychological view of 
James-Lange (James, 1884) implied that the two 
were the same. However, philosophers argued that 
emotions are not just bodily sensations; the two 
have different objects. Bodily sensations are about 
awareness of the internal state of the body. Emo- 
tional feelings are directed toward objects in the 
external world. Neuroscientific evidence based on 
functional magnetic resonance imaging (fMRI) 
tends to provide important validation of the theo- 
retical view of James-Lange that neural systems 
supporting the perception of bodily states provide 
a fundamental ingredient for the subjective experi- 
ence of emotions. This is consistent with contem- 
porary neuroscientific views (e.g., see Craig, 2002), 
which suggest that the anterior insular cortex, es- 
pecially on the right side of the brain, plays an im- 
portant role in the mapping of bodily states and 
their translation into emotional feelings. The view 
of Damasio (1999, 2003) is consistent with this 
notion, but it suggests further that emotional feel- 
ings are not just about the body, but they are also 



about things in the world as well. In other words, 
sensing changes in the body would require neural 
systems, of which the anterior insular cortex is a 
critical substrate. However, the feelings that ac- 
company emotions require additional brain re- 
gions. In Damasio's view, feelings arise in conscious 
awareness through the representation of bodily 
changes in relation to the object or event that in- 
cited the bodily changes. This second-order map- 
ping of the relationship between organism and 
object occurs in brain regions that can integrate in- 
formation about the body with information about 
the world. Such regions include the anterior cingu- 
late cortex (figure 12.1), especially its dorsal part. 

According to Damasio (1994, 1999, 2003), there 
is an important distinction between emotions and 
feelings. Emotions are a collection of changes in bod- 
ily and brain states triggered by a dedicated brain 
system that responds to the content of one's percep- 
tions of a particular entity or event. The responses 
toward the body proper enacted in a bodily state in- 
volve physiological modifications that range from 
changes that are hidden from an external observer 
(e.g., changes in heart rate, smooth muscle contrac- 
tion, endocrine release, etc.) to changes that are per- 
ceptible to an external observer (e.g., skin color, 
body posture, facial expression, etc.). The signals 
generated by these changes toward the brain itself 
produce changes that are mostly perceptible to the 
individual in whom they were enacted, which then 
provide the essential ingredients for what is ulti- 
mately perceived as a feeling. Thus emotions are what 
an outside observer can see, or at least can measure 
through neuroscientific tools. Feelings are what the 
individual senses or subjectively experiences. 

An emotion begins with appraisal of an emo- 
tionally competent object. An emotionally compe- 
tent object is basically the object of one's emotion, 
such as the person you are in love with. In neural 
terms, images related to the emotional object are 
represented in one or more of the brain's sensory 
processing systems. Regardless of how short this 
presentation is, signals related to the presence of 
that object are made available to a number of 
emotion-triggering sites elsewhere in the brain. 
Some of these emotion-triggering sites are the 
amygdala and the orbitofrontal cortex (see figure 
12.1). Evidence suggests that there may be some 
difference in the way the amygdala and the or- 
bitofrontal cortex process emotional information: 
The amygdala is more engaged in the triggering of 
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emotions when the emotional object is present in 
the environment; the orbitofrontal cortex is more 
important when the emotional object is recalled 
from memory (Bechara, Damasio, & Damasio, 
2003; see also chapter 11, this volume). 

In order to create an emotional state, the activ- 
ity in triggering sites must be propagated to execu- 
tion sites by means of neural connections. The 
emotion execution sites are visceral motor structures 
that include the hypothalamus, the basal forebrain, 



and some nuclei in the brain stem tegmentum (fig- 
ure 12.1). 

Feelings result from neural patterns that repre- 
sent changes in the body's response to an emotional 
object. Signals from body states are relayed back to 
the brain, and representations of these body states 
are formed at the level of visceral sensory nuclei in 
the brain stem. Representations of these body sig- 
nals also form at the level of the insular cortex and 
lateral somatosensory cortex (figure 12.1). It is 
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Figure 1 2.1 . Information related to the emotionally competent object is represented in one or more of the brains 
sensory processing systems. This information, which can be derived from the environment or recalled from mem- 
ory, is made available to the amygdala and the orbitofrontal cortex, which are trigger sites for emotion. The emo- 
tion execution sites include the hypothalamus, the basal forebrain, and nuclei in the brain stem tegmentum. Only 
the visceral response is represented, although emotion comprises endocrine and somatomotor responses as well. 
Visceral sensations reach the anterior insular cortex by passing through the brain stem. Feelings result from the re- 
representation of changes in the viscera in relation to the object or event that incited them. The anterior cingulate 
cortex is a site where this second-order map is realized. 
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most likely that the reception of bodily signals at 
the level of the brain stem does not give rise to con- 
scious feelings as we know them, but the reception 
of these signals at the level of the cortex does so. 
The anterior insular cortex plays a special role in 
mapping visceral states and in bringing interocep- 
tive signals to conscious perception. It is less clear 
whether the anterior insular cortex also plays a spe- 
cial role in translating the visceral states into subjec- 
tive feeling and self-awareness. In Damasios view, 
feelings arise in conscious awareness through the 
representation of bodily changes in relation to the 
emotional object (present or recalled) that incited 
the bodily changes. A first-order mapping of self is 
supported by structures in the brain stem, insular 
cortex, and somatosensory cortex. However, addi- 
tional regions, such as the anterior cingulate cortex, 
are required for a second-order mapping of the rela- 
tionship between organism and emotional object, 
and the integration of information about the body 
with information about the world. 



Disturbances of Emotional Experience 
after Focal Brain Damage 

There are many instances of disturbance in emo- 
tions and feelings linked to focal lesions to struc- 
tures outlined earlier. The following is a review of 
evidence from neurological patients with focal 
brain damage, as well as supporting functional 
neuroimaging evidence, demonstrating the role of 
these specific neural structures in processing infor- 
mation about emotions, feelings, and social behav- 
ior. The specific neural structures addressed are the 
amygdala, the insular and somatosensory cortex, 
and the orbitofrontal and anterior cingulate cortex. 

Amygdala Damage 

Clinical observations of patients with amygdala 
damage (especially when the damage is bilateral; 
figure 12.2) reveal that these patients express one 
form of emotional lopsidedness: Negative emotions 




Figure 12.2. (a) Coronal sections through the amygdala taken from the 3-D reconstruction of brains of patients 
with bilateral amygdala damage. The region showing bilateral amygdala damage is highlighted by circles, (b) Coro- 
nal sections through the brain of a patient suffering from anosognosia. These coronoal sections show extensive 
damage in the right parietal region that include the insula and somatosensory cortices (SI1, SI). The left parietal re- 
gion is intact, (c) Left midsagittal (left), inferior (center), and right midsagittal (right) views of the brain of a patient 
with bilateral damage to the ventromedial region of the prefrontal cortex. 
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such as anger and fear are less frequent and less in- 
tense in comparison to positive emotions (Damasio, 
1999). Many laboratory experiments have also es- 
tablished problems in these patients with process- 
ing emotional information, especially in relation to 
fear (Adolphs, Tranel, & Damasio, 1998; Adolphs, 
Tranel, Damasio, & Damasio, 1995; LaBar, LeDoux, 
Spencer, & Phelps, 1995; Phelps et al., 1998). Note- 
worthy, at least when the damage occurs earlier in 
life, is that these patients grow up to have many 
abnormal social behaviors and functions (Adolphs 
et al., 1995; Tranel & Hyman, 1990). 

Laboratory experiments suggest that the amyg- 
dala is a critical substrate in the neural system nec- 
essary for the triggering of emotional states from 
what we have termed primary inducers (Bechara et 
al., 2003). Primary inducers are stimuli or entities 
that are innate or learned to be pleasant or aversive. 
Once they are present in the immediate environ- 
ment, they automatically, quickly, and obligatorily 
elicit an emotional response. Examples of primary 
inducers include the encounter of a feared object 
such as a snake. Winning or losing a large sum of 
money, as in the case of being told that you won the 
lottery, is a type of learned information, but it has 
the property of instantly, automatically, and obliga- 
torily eliciting an emotional response. This is also 
an example of a primary inducer. Secondary induc- 
ers, on the other hand, are entities generated by the 
recall of a personal or hypothetical emotional event 
(i.e., thoughts and memories about the primary in- 
ducer), which, when they are brought to working 
memory, slowly and gradually begin to elicit an 
emotional response. Examples of secondary induc- 
ers include the emotional response elicited by the 
memory of encountering or being bitten by a snake, 
the memory or the imagination of winning the lot- 
tery, and the recall or imagination of the death of a 
loved one. 

Several lines of study suggest that the amyg- 
dala is a critical substrate in the neural system nec- 
essary for the triggering of emotional states from 
primary inducers. Patients with bilateral amygdala 
lesions have reduced, but not completely blocked, 
autonomic reactivity to startlingly aversive loud 
sounds (Bechara, Damasio, Damasio, & Lee, 
1999). These patients also do not acquire condi- 
tioned autonomic responses to the same aversive 
loud sounds, even when the damage is unilateral 
(Bechara et al., 1995; LaBar et al., 1995). Amygdala 



lesions in humans have also been shown to reduce 
autonomic reactivity to a variety of stressful stimuli 
(Leeetal., 1988, 1998). 

Bilateral amygdala damage in humans also in- 
terferes with the emotional response to cognitive in- 
formation that through learning has acquired 
properties that automatically and obligatorily elicit 
emotional responses. Examples of this cognitive in- 
formation are learned concepts such as winning or 
losing. The announcement that you have won a No- 
bel Prize, an Oscar award, or the lottery can in- 
stantly, automatically, involuntarily, and obligatorily 
elicit an emotional response. Emotional reactions to 
gains and losses of money, for example, are learned 
responses because we were not born with them. 
However, through development and learning, these 
reactions become automatic. We do not know how 
this transfer occurs. However, we have presented 
evidence showing that patients with bilateral amyg- 
dala lesions failed to trigger emotional responses in 
reaction to the winning or losing of various 
amounts of money (Bechara et al., 1999). 

The results of functional neuroimaging studies 
corroborate those from lesion studies. For instance, 
activation of the amygdala has been shown in classi- 
cal conditioning experiments (LaBar, Gatenby Gore, 
LeDoux, & Phelps, 1998). Other functional neu- 
roimaging studies have revealed amygdala activation 
in reaction to winning and losing money (Zalla et 
al., 2000). Also interesting is that humans tend to 
automatically, involuntarily, and obligatorily elicit a 
pleasure response when they solve a puzzle or un- 
cover a solution to a logical problem. In functional 
neuroimaging experiments involving asking human 
subjects to find solutions to a series of logical prob- 
lems, amygdala activations were associated with the 
"aha" in reaction to finding the solution to a given 
logical problem (Parsons & Oshercon, 2001). 

In essence, the amygdala links the features of a 
stimulus with the expressed emotional or affective 
value of that stimulus (Malkova, Gaffan, & Murray, 
1997). However, the amygdala appears to respond 
only when the stimuli are actually present in the 
environment (Whalen, 1998). 

Damage to the Insular or 
Somatosensory Cortex 

The classical clinical condition of patients with pari- 
etal damage (involving the insular, somatosensory, 
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and adjacent cortex), especially on the right side, 
which demonstrates alterations in emotional expe- 
rience, is called anosognosia. Anosognosia means 
denial of illness or failure to recognize an illness 
(see figure 12.2). The condition is characterized by 
apathy and placidity. It is most commonly seen in 
association with right-hemisphere lesions (as op- 
posed to left). 

The classical example of this condition is that 
the patient is paralyzed in the left side of the 
body, unable to move hand, arm, and leg, and un- 
able to stand or walk. When asked how they feel, 
patients with anosognosia report that they feel 
fine, and they seem oblivious to the entire prob- 
lem. In stroke patients, the unawareness is typi- 
cally most profound during the first few days after 
onset. In a few days or a week, patients will begin 
to acknowledge that they have suffered a stroke 
and that they are weak or numb, but they mini- 
mize the implications of the impairment. In the 
chronic epoch (3 months or more after onset), the 
patients may provide a more accurate account of 
their physical disabilities. However, defects in the 
appreciation of acquired cognitive limitations 
may persist for months or years. Patients with 
similar damage on the left side of their brain are 
usually cognizant of their deficit and often feel 
depressed. 

Many laboratory experiments have established 
problems with processing emotional information 
in these patients, such as empathy and recognition 
of emotions in facial expressions (Adolphs, Dama- 
sio, Tranel, & Damasio, 1996). Furthermore, al- 
though the paralysis and neurological handicaps of 
these patients limit their social interactions and 
mask potential abnormal social behaviors, in- 
stances in which these patients were allowed ex- 
tensive social interactions revealed that patients 
with this condition exhibit severe impairments in 
judgment and failure to observe social convention. 
One illustrative example is the case of the Supreme 
Court Justice William O. Douglas described by 
Damasio in his book Descartes' Error (Damasio, 
1994). 

Support for the idea that the insular and 
somatosensory cortex are parts of a neural system 
that subserves emotions and feelings also comes 
from numerous experiments using functional neu- 
roimaging methods (Dolan, 2002). In addition, 
evidence from functional neuroimaging studies 



suggests that beside the insular and somatosensory 
cortex, neighboring regions that include the pos- 
terior cingulate cortex are consistently activated 
in experiments involving the generation of feeling 
states (Damasio et al, 2000; Maddock, 1999), 
which suggests that the whole region plays a role in 
the generation of feelings from autobiographical 
memory. 

Lesions of the Orbitofrontal and Anterior 
Cingulate Cortex 

Patients with orbitofrontal cortex damage exhibit 
varying degrees of disturbance in emotional experi- 
ence, depending on the location and extent of the 
damage (figure 12.2). If the damage is localized, 
especially in the more anterior sector of the or- 
bitofrontal region (i.e., toward the front of the 
brain), the patients exhibit many manifestations in- 
cluding alterations of emotional experience and 
social functioning. Previously well-adapted indi- 
viduals become unable to observe social conven- 
tions and decide advantageously on personal 
matters, and their ability to express emotion and 
to experience feelings appropriately in social sit- 
uations becomes compromised (Bechara, Damasio, 
& Damasio, 2000; Bechara, Tranel, & Damasio, 
2002). If the damage is more extensive, especially 
when it involves parts of the anterior cingulate, the 
patients exhibit additional problems in impulse 
control, disinhibition, and antisocial behavior. For 
instance, such patients may utter obscene words, 
make improper sexual advances, or say the first 
thing that comes to mind, without considering the 
social correctness of what they say or do. As an ex- 
ample, some of these patients may urinate in a 
completely inappropriate social setting, when the 
urge arises, without any regard to the social rules of 
decency. 

With more extensive damage, the patient may 
suffer a condition known as akinetic mutism, espe- 
cially when the damage involves most of the ante- 
rior cingulate cortex and a surrounding region 
called the supplementary motor area. The condi- 
tion is a combination of mutism and akinesia. The 
lesions may result from strokes related to impair- 
ment of blood supply in the anterior cerebral artery 
territories and, in some cases, from rupture of 
aneurysms of the anterior communicating artery or 
anterior cerebral artery. It may also result from 



184 



Perception, Cognition, and Emotion 



parasagittal tumors (e.g., meningiomas of the falx 
cerebri). The lesion can be unilateral or bilateral. 
There is no difference between left- and right-side 
lesions in terms of causing the condition. The dif- 
ference between unilateral and bilateral lesions ap- 
pears to be only in relation to course of recovery: 
With unilateral lesions, the condition persists for 
1 to 2 weeks; with bilateral lesions, the condition 
may persist for many months. The patient with 
akinetic mutism makes no effort to communicate 
verbally or by gesture. Movements are limited to 
the eyes (tracking moving targets) and to body or 
arm movements connected with daily necessities 
(eating, pulling bed sheets, getting up to go to the 
bathroom). Speech is exceptionally sparse, with 
only rare isolated utterances, but linguistically cor- 
rect and well articulated (although generally hypo- 
phonic). With extensive prompting, the patient 
may repeat words and short sentences. 

Provided that the amygdala, insular and so- 
matosensory cortices were normal during develop- 
ment, emotional states associated with secondary 
inducers develop normally. Generating emotional 
states from secondary inducers depends on cortical 
circuitry in which the orbitofrontal cortex plays 
a central role. Evidence suggests that the or- 
bitofrontal region is a critical substrate in the neu- 
ral system necessary for the triggering of emotional 
states from secondary inducers, that is, from re- 
calling or imagining an emotional event (Bechara 
etal.,2003). 



Development of the Neural Systems 
Subserving Emotions and Feelings 

While the amygdala is engaged in emotional situa- 
tions requiring a rapid response, that is, low-order 
emotional reactions arising from relatively auto- 
matic processes (LeDoux, 1996), the orbitofrontal 
cortex is engaged in emotional situations driven by 
thoughts and reason. Once this initial amygdala 
emotional response is over, high-order emotional 
reactions begin to arise from relatively more con- 
trolled, higher-order processes involved in thinking, 
reasoning, and consciousness. Unlike the amygdala 
response, which is sudden and habituates quickly, 
the orbitofrontal cortex response is deliberate and 
slow, and lasts for a long time. 

Thus the orbitofrontal cortex helps predict 
the emotion of the future, thereby forecasting the 



consequences of one's own actions. However, it is 
important to note that the amygdala system is a 
priori a necessary step for the normal development 
of the orbitofrontal system for triggering emotional 
states from secondary inducers (i.e., from thoughts 
and reflections). The normal acquisition of second- 
ary inducers requires the integrity of the amygdala, 
and also the insular and somatosensory cortex. 
When the amygdala, or critical components of the 
insular and somatosensory cortex, is damaged, then 
primary inducers cannot induce emotional states. 
Furthermore, signals from triggered emotions can- 
not be transmitted to the insular and somatosen- 
sory cortex and then get translated into conscious 
feelings. For instance, individuals with a congenital 
absence of a specific type of neurons specialized for 
transmitting pain signals from the skin, called C 
fibers, do not feel pain, and they are unable to con- 
struct feeling representations related to pain. It fol- 
lows that such individuals are unable to fear 
situations that lead to pain, or empathize in con- 
texts related to pain; that is, they lack the brain 
representations of what it feels like to be in pain 
(Damasio, 1994, 1999). Thus the normal develop- 
ment of the orbitofrontal system (which is important 
for triggering emotions from secondary inducers) is 
contingent upon integrity of the amygdala system, 
which is critical for triggering emotions from pri- 
mary inducers. 

Given this neural framework, it follows that 
there may be a fundamental difference between two 
types of abnormalities that lead to distorted brain 
representations of emotional and feeling states, 
which in turn lead to abnormal cognition and be- 
havior, especially in the area of judgment and deci- 
sion making. The following paragraphs outline of 
the nature of these potential abnormalities. 

One abnormality is neurobiological in nature 
and may relate to (1) abnormal receptors or cells 
concerned with the triggering or detection of emo- 
tional signals at the level of the viscera and internal 
milieu; (2) abnormal peripheral neural and en- 
docrine systems concerned with transmission of 
emotional signals from the viscera and internal mi- 
lieu to the brain stem, that is, the spinal cord, the 
vagus nerve, and the circumventricular organs 
(the brain areas that lack a blood-brain barrier); or 
(3) abnormal neural systems involved in the trig- 
gering (e.g., the amygdala, orbitofrontal cortex, 
and effector structures in the brain stem) or build- 
ing of representations of emotional or feeling states 
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(e.g., sensory nuclei in the brain stem, and insular 
or somatosensory cortex). 

The other abnormality is environmental in na- 
ture and relates to social learning. For instance, 
growing up in a social environment where, say, 
killing another individual is glorified and encour- 
aged leads to abnormal development of the repre- 
sentations of the emotional or feeling states 
associated with the act of killing. Although both 
abnormalities may be difficult to distinguish from 
each other at a behavioral level, the two are distin- 
guishable at a physiological level. 

We argue that individuals with abnormal social 
learning are capable of triggering emotional states 
under a variety of laboratory conditions. These in- 
dividuals have the capacity to empathize, feel re- 
morse, and fear negative consequences. In contrast, 
individuals with neurobiological abnormalities 
demonstrate failure to trigger emotional states un- 
der the same laboratory conditions. Such individu- 
als cannot express emotions, empathize, or fear 
negative consequences. The distinction between the 
two abnormalities has important social and legal 
implications. Individuals whose abnormal neural 
representations of emotional or feeling states relate 
to faulty social learning can reverse this abnormality 
and unlearn the antisocial behavior once they are 
exposed to proper learning contingencies. In other 
words, these individuals are likely to benefit from 
cognitive and behavioral rehabilitation. In contrast, 
individuals with underlying neurobiological abnor- 
malities do not have the capacity to reverse these 
emotional or feeling abnormalities. Consequently, 
these individuals demonstrate repeated and persis- 
tent failures to learn from previous mistakes, even 
in the face of rising and severe punishment. It fol- 
lows that these individuals are unlikely to benefit 
from rehabilitation. 



The Interplay between Emotions, 
Feelings, and Decision Making 

Situations involving personal and social matters are 
strongly associated with positive and negative emo- 
tions. Reward or punishment, pleasure or pain, 
happiness or sadness all produce changes in bodily 
states, and these changes are expressed as emotions. 
We argue that such prior emotional experiences of- 
ten come into play when we are deliberating a deci- 
sion. Whether these emotions remain unconscious 



or are perceived consciously in the form of feelings, 
they provide the go, stop, and turn signals needed 
for making advantageous decisions. In other words, 
the activation of these brain representations of emo- 
tional and body states provides biasing signals that 
covertly or overtly mark various options and sce- 
narios with a value. Accordingly, these biases assist 
in the selection of advantageous responses from 
among an array of available options. Deprived of 
these biases, response options become more or less 
equalized, and decisions become dependent on a 
slow reasoned cost-benefit analysis of numerous 
and often conflicting options. At the end, the result 
is an inadequate selection of a response. 

Phineas Gage: A Brief History 

Phineas Gage was a dynamite worker who survived 
an explosion that blasted an iron-tamping bar 
through the front of his head. Before the accident, 
Phineas Gage was a man of normal intelligence, re- 
sponsible, sociable, and popular among peers and 
friends. After the accident, his recovery was re- 
markable. He survived this accident with normal 
intelligence, memory, speech, sensation, and move- 
ment. However, his behavior changed completely: 
He became irresponsible and untrustworthy, impa- 
tient of restraint or advice when it conflicted with 
his desires (Damasio, 1994). Phineas Gage died and 
an autopsy was not performed to determine the lo- 
cation of his brain lesion. However, the skull of 
Phineas Gage was preserved and kept at a museum 
at Harvard University. Using modern neuroimag- 
ing techniques, Hanna Damasio and colleagues at 
the University of Iowa reconstructed the brain of 
Phineas Gage. Based on measures taken from his 
skull, they reconstituted the path of the iron bar 
and determined the most likely location of his brain 
lesion (Damasio, Grabowski, Frank, Galburda, & 
Damasio, 1994). The key finding of this neuroimag- 
ing study was that the most likely placement of 
Gage's lesion was the ventromedial (VM) region of 
the prefrontal cortex on both sides. The damage 
was relatively extensive and involved considerable 
portions of the anterior cingulate. 

Over the years, we have studied numerous pa- 
tients with VM lesions. Such patients develop se- 
vere impairments in personal and social decision 
making, in spite of otherwise largely preserved in- 
tellectual abilities. These patients were intelligent 
and creative before their brain damage. After the 
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damage, they had difficulties planning their work- 
day and future and difficulties in choosing friends, 
partners, and activities. The actions they elect to 
pursue often lead to diverse losses, such as financial 
losses, losses in social standing, losses of family and 
friends. The choices they make are no longer ad- 
vantageous and are remarkably different from the 
kinds of choices they were known to make before 
their brain injuries. These patients often decide 
against their best interests. They are unable to learn 
from previous mistakes, as reflected by repeated 
engagement in decisions that lead to negative conse- 
quences. In striking contrast to this real-life decision- 
making impairment, the patients perform normally 
in most laboratory tests of problem solving. Their 
intellect remains normal, as measured by conven- 
tional clinical neuropsychological tests. 

The Somatic Marker Hypothesis 

While these VM patients were intact on most neu- 
ropsychological tests, there were abnormalities in 
emotion and feeling, along with the abnormalities 
in decision making. Based on these observations, 
the somatic marker hypothesis was proposed (Dama- 
sio, 1994), which posits that the neural basis of the 
decision-making impairment characteristic of pa- 
tients with VM prefrontal lobe damage is defective 
activation of somatic states (emotional signals) that 
attach value to given options and scenarios. These 
emotional signals function as covert, or overt, biases 
for guiding decisions. Deprived of these emotional 
signals, patients must rely on slow cost-benefit 
analyses of various conflicting options. These op- 
tions may be too numerous, and their analysis may 
be too lengthy to permit rapid, online decisions to 
take place appropriately. Patients may resort to de- 
ciding based on the immediate reward of an op- 
tion, or may fail to decide altogether if many options 
have the same basic value. 

In essence, when we make decisions, mecha- 
nisms of arousal, attention, and memory are neces- 
sary to evoke and display the representations of 
various options and scenarios in our mind's eye. 
However, another mechanism is necessary for 
weighing these various options and for selecting the 
most advantageous response. This mechanism for 
selecting good from bad is what we call decision- 
making, and the physiological changes occurring in 
association with the behavioral selection are part of 
what we call somatic states (or somatic signaling). 



Evidence That Emotion 
Guides Decisions 

Situations involving personal and social matters are 
strongly associated with positive and negative emo- 
tions. Reward or punishment, pleasure or pain, 
happiness or sadness all produce changes in bodily 
states, and these changes are expressed as emotions. 
We believe that such prior emotional experiences 
often come into play when we are deliberating a 
decision. Whether these emotions remain uncon- 
scious or are perceived consciously in the form 
of feelings, they provide the go, stop, and turn sig- 
nals needed for making advantageous decisions. In 
other words, the activation of these somatic states 
provides biasing signals that covertly or overtly 
mark various options and scenarios with a value. 
Accordingly, these biases assist in the selection of 
advantageous responses from among an array of 
available options. Deprived of these biases or so- 
matic markers, response options become more or 
less equalized and decisions become dependent on 
a slow reasoned cost-benefit analysis of numerous 
and often conflicting options. At the end, the result 
is an inadequate selection of a response. We con- 
ducted several studies that support the idea that de- 
cision making is a process guided by emotions. 

The Iowa Gambling Task 

For many years, these VM patients presented a 
puzzling defect. Although the decision-making im- 
pairment was obvious in the real-world behavioral 
lives of these patients, there was no effective labo- 
ratory probe to detect and measure this impair- 
ment. For this reason, we developed what became 
known as the Iowa gambling task, which enabled 
us to detect these patients' elusive impairment in 
the laboratory, measure it, and investigate its pos- 
sible causes (Bechara, Damasio, Damasio, & An- 
derson, 1994). The gambling task mimics real-life 
decisions closely. The task is carried out in real 
time and it resembles real-world contingencies. It 
factors reward and punishment (i.e., winning and 
losing money) in such a way that it creates a con- 
flict between an immediate, luring reward and a 
delayed, probabilistic punishment. Therefore, the 
task engages the subject in a quest to make advan- 
tageous choices. As in real-life choices, the task 
offers choices that may be risky, and there is no ob- 
vious explanation of how, when, or what to choose. 



The Neurology of Emotions and Feelings 



187 



Each choice is full of uncertainty because a precise 
calculation or prediction of the outcome of a given 
choice is not possible. The way that one can do 
well on this task is to follow one's hunches and gut 
feelings. 

More specifically, this task involves four decks 
of cards. The goal in the task is to maximize profit 
on a loan of play money. Subjects are required to 
make a series of 100 card selections. However, they 
are not told ahead of time how many card selec- 
tions they are going to make. Subjects can select 
one card at a time from any deck they choose, and 
they are free to switch from any deck to another at 
any time, and as often as they wish. However, the 
subject's decision to select from one deck versus 
another is largely influenced by various schedules 
of immediate reward and future punishment. 
These schedules are preprogrammed and known to 
the examiner, but not to the subject, and they en- 
tail the following principles: Every time the subject 
selects a card from two decks (A and B), the subject 
gets $100. Every time the subject selects a card 
from the two other decks (C or D), the subject 
gets $50. However, in each of the four decks, sub- 
jects encounter unpredictable punishments (money 
loss). The punishment is set to be higher in the 
high-paying Decks A and B, and lower in the low- 
paying Decks C and D. For example, if 10 cards 
were picked from Deck A, one would earn $1,000. 
However, in those 10 card picks, 5 unpredictable 
punishments would be encountered, ranging from 
$150 to $350, bringing a total cost of $1,250. Deck 
B is similar: Every 10 cards that were picked from 
Deck B would earn $1,000; however, these 10 card 
picks would encounter one high punishment of 
$1,250. On the other hand, every 10 cards from 
Decks C or D earn only $500, but only cost $250 
in punishment. Hence, Decks A and B are disad- 
vantageous because they cost more in the long run; 
that is, one loses $250 every 10 cards. Decks C and 
D are advantageous because they result in an over- 
all gain in the long run; that is, one wins $250 
every 10 cards. 

We investigated the performance of normal 
controls and patients with VM prefrontal cortex le- 
sions on this task. Normal subjects avoided the bad 
decks A and B and preferred the good decks C and 
D. In sharp contrast, the VM patients did not avoid 
the bad decks A and B; indeed, they preferred 
decks A and B. From these results, we suggested 
that the patients' performance profile is comparable 



to their real-life inability to decide advantageously. 
This is especially true in personal and social mat- 
ters, a domain for which in life, as in the task, an 
exact calculation of future outcomes is not possible 
and choices must be based on hunches and gut 
feelings. 

Emotional Signals Guide Decisions 

In light of the finding that the gambling task is an 
instrument that detects the decision-making impair- 
ment of VM patients in the laboratory, we went on 
to address the next question of whether the impair- 
ment is linked to a failure in somatic (emotional) 
signaling (Bechara, Tranel, Damasio, & Damasio, 
1996). 

To address this question, we added a physio- 
logical measure to the gambling task. The goal was 
to assess somatic state activation (or generation of 
emotional signals) while subjects were making de- 
cisions during the gambling task. We studied two 
groups: normal subjects and VM patients. We had 
them perform the gambling task while we recorded 
their electrodermal activity (skin conductance re- 
sponse, SCR). As the body begins to change after a 
thought, and as a given emotion begins to be en- 
acted, the autonomic nervous system begins to 
increase the activity in the skin's sweat glands. Al- 
though this sweating activity is relatively small and 
not observable by the naked eye, it can be ampli- 
fied and recorded by a polygraph as a wave. The 
amplitude of this wave can be measured and thus 
provide an indirect measure of the emotion experi- 
enced by the subject. 

Both normal subjects and VM patients gener- 
ated SCRs after they had picked a card and were 
told that they won or lost money. The most impor- 
tant difference, however, was that normal subjects, 
as they became experienced with the task, began to 
generate SCRs prior to the selection of any cards, 
that is, during the time when they were pondering 
from which deck to choose. These anticipatory 
SCRs were more pronounced before picking a card 
from the risky decks A and B, when compared to 
the safe decks C and D. In other words, these an- 
ticipatory SCRs were like gut feelings that warned 
the subject against picking from the bad decks. 
Frontal patients failed to generate such SCRs before 
picking a card. This failure to generate anticipatory 
SCRs before picking cards from the bad decks cor- 
relates with their failure to avoid these bad decks 
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and choose advantageously in this task. These re- 
sults provide strong support for the notion that de- 
cision making is guided by emotional signals (gut 
feelings) that are generated in anticipation of future 
events. 

Emotional Signals Do Not Need 
to Be Conscious 

Further experiments revealed that these biasing 
somatic signals (gut feelings) do not need to be 
perceived consciously. We carried out an experi- 
ment similar to the previous one, in which we 
tested normal subjects and VM patients on the 
gambling task while recording their SCRs. How- 
ever, every time the subject picked 10 cards from 
the decks, we would stop the game briefly and ask 
subjects to declare whatever they knew about 
what was going on in the game (Bechara, Damasio, 
Tranel, & Damasio, 1997). From the answers to 
the questions, we were able to distinguish four pe- 
riods as subjects went from the first to the last trial 
in the task. The first was a prepunishment period, 
when subjects sampled the decks, and before they 
had yet encountered any punishment. The second 
was a prehunch period, when subjects began to 
encounter punishment, but when asked about 
what was going on in the game, they had no clue. 
The third was a hunch period, when subjects be- 
gan to express a hunch about which decks were 
riskier but were not sure. The fourth was a con- 
ceptual period, when subjects knew very well the 
contingencies in the task, and which decks were 
the good ones and which were bad ones, and why 
this was so. 

When examining the anticipatory SCRs from 
each period, we found that there was no significant 
activity during the prepunishment period. These 
were expected results because, at this stage, the 
subjects were picking cards and gaining money, and 
had not encountered any losses yet. Then there was 
a substantial rise in anticipatory responses during 
the prehunch period, that is, after encountering 
some money losses, but still before the subject had 
any clue about what was going on in the game. This 
SCR activity was sustained for the remaining peri- 
ods, that is, during the hunch and then during the 
conceptual period. When examining the behavior 
during each period, we found that there was a pref- 
erence for the high-paying decks (A and B) during 
the prepunishment period. Then there was a hint of 



a shift in the pattern of card selection, away from 
the bad decks, even in the prehunch period. This 
shift in preference for the good decks became more 
pronounced during the hunch and conceptual peri- 
ods. The VM patients, on the other hand, never re- 
ported a hunch about which of the decks were 
good or bad. Furthermore, they never developed 
anticipatory SCRs, and they continued to choose 
more cards from the bad decks A and B relative to 
the good decks C and D. 

An especially intriguing observation was that 
not all the normal control subjects were able to fig- 
ure out the task, explicitly, in the sense that they 
did not reach the conceptual period. Only 70% of 
them were able to do so. Although 30% of controls 
did not reach the conceptual period, they still per- 
formed advantageously. On the other hand, 50% of 
the VM patients were able to reach the conceptual 
period and state explicitly which decks were good 
and which ones were bad and why. Although 50% 
of the VM patients did reach the conceptual pe- 
riod, they still performed disadvantageously. After 
the experiment, these VM patients were confronted 
with the question, Why did you continue to pick 
from the decks you thought were bad? The patients 
would resort to excuses such as "I was trying to fig- 
ure out what happens if I kept playing the $100 
decks" or "I wanted to recover my losses fast, and 
the $50 decks are too slow." 

These results show that VM patients continue 
to choose disadvantageously in the gambling task, 
even after realizing explicitly the consequences of 
their action. This suggests that the anticipatory 
SCRs represent unconscious biases derived from 
prior experiences with reward and punishment. 
These biases (or gut feelings) help deter the normal 
subject from pursuing a course of action that is dis- 
advantageous in the future. This occurs even be- 
fore subjects become aware of the goodness or 
badness of the choice they are about to make. 
Without these biases, the knowledge of what is 
right and what is wrong may still become available. 
However, by itself, this knowledge is not sufficient 
to ensure an advantageous behavior. Therefore, 
although VM patients may manifest declarative 
knowledge of what is right and what is wrong, they 
fail to act accordingly. The VM patients may say the 
right thing, but they do the wrong thing. 

Thus, knowledge without emotion or somatic 
signaling leads to dissociation between what one 
knows or says and how one decides to act. This 
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dissociation is not restricted to neurological pa- 
tients but also applies to neuropsychiatric condi- 
tions with suspected pathology in the VM cortex or 
other components of the neural circuitry that pro- 
cess emotion. Addiction is one example, where pa- 
tients know the consequences of their drug-seeking 
behavior but still take the drug. Psychopathy is an- 
other example, where psychopaths can be fully 
aware of the consequences of their actions but still 
go ahead and plan the killing or rape of a victim. 



A Brain-Based Model 
of Robot Decisions 

There are many implications for neuroergonomics 
of the findings on emotion and decision making 
discussed in this chapter. One area that is also dis- 
cussed elsewhere in this volume is affective robot- 
ics. Breazeal and Picard (chapter 18, this volume) 
provide a compelling robot model that incorpo- 
rates emotions and social interactions. This is a re- 
markable advance, since their approach is based on 
evolutionary and psychological studies of emo- 
tions, and the model yields exciting results. I pro- 
pose a brain-based model of robot decisions based 
on what we know about brain mechanisms of emo- 
tions as reviewed in this chapter. Obviously, some 
of the steps of my proposed model overlap with 
those of Breazeal and Picard's model. However, 
some differ, especially in the process that Breazeal 
and Picard referr to as the cognitive-affective con- 
trol system. 

Although most researchers on emotions are 
preoccupied with understanding subtle differences 
between states such as anger, sadness, fear, and 
other different shades of emotions, I think that the 
most crucial information about emotions that we 
need at this stage is the following: (1) is the emo- 
tion positive or negative; and (2) is it mild, moder- 
ate, or strong? This is a classification that is similar 
and agrees with that of Breazeal and Picard. 

The first question we need to know is how hu- 
mans (and in this case robots) assign value to op- 
tions. Here the term value is interchangeable with 
the terms emotion, feeling, motivation, affect, or so- 
matic marker. The idea is that the whole purpose of 
emotion is to assign value for every option we 
have. For example, the value of a drink of water to 
a person stranded in a desert is different from its 
value to a person who just had a super-size Pepsi. 



Other factors that affect the value of a choice in- 
clude time, which explains a phenomenon named 
temporal discounting, that is, why information con- 
veying immediacy (e.g., having a heart attack to- 
morrow) exerts a stronger influence on decisions 
than information conveying delayed outcomes (e.g., 
having heart attack 20 years from now). Bechara 
and Damasio (2005) described elsewhere in more 
detail how such factors may be implemented in the 
brain. Neuroscientists have been able to address 
the question of how the brain can encode the value 
of various options on a common scale (Montague 
& Berns, 2002), thus suggesting that there may be 
a common neural currency that encodes the value 
of different options. This may, for example, allow 
the reward value of money to be compared to that 
of food, sex or other goods. 

The Primacy of Emotion 
during Development 

A key to emotional development is that the amyg- 
dala system (primary induction) and the insular or 
somatosensory system (which holds records of 
every emotional experience triggered by the amyg- 
dala) must be healthy, in the first place, in order for 
the prefrontal system to develop and function nor- 
mally, that is, assign appropriate value to a given 
option. In terms of a robot, we can build this sys- 
tem that holds records of values linked to categories 
of actions using one of two methods. First, build a 
robot with some values that are innately assigned, 
that is, a system that simulates the amygdala sys- 
tem, which responds to primary inducers, and then 
let the robot execute a whole bunch of behaviors 
and decisions, with each one being followed by a 
consequence (i.e., reward or punishment of mild, 
moderate, or strong magnitude). In this way, the ro- 
bot can build a registry of values connected to par- 
ticular actions, that is, equivalent to the insular or 
somatosensory and prefrontal systems for second- 
ary induction. In other words, here we have to as- 
sume that the robot is like a newly born child 
exposed to the world. Learning begins from scratch, 
and building a value system is a process similar to 
that of raising a child, especially in terms of rein- 
forcing good behaviors and punishing bad ones. 
The other method is to bypass this learning process 
and install into a robot a registry of values based on 
what we already know from normal human devel- 
opment. The latter method is somewhat inaccurate 
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because it does not take into account human indi- 
viduality, which differs from person to person be- 
cause of their different developmental histories. At 
best, this method may account for an average, 
rather than an individual, human behavior. 

The Willpower to Endure Sacrifices 
and Resist Temptations 

Willpower, as defined by the Encarta World English 
Dictionary, is a combination of determination and 
self-discipline that enables somebody to do some- 
thing despite the difficulties involved. This is the 
mechanism that enables one to endure sacrifices 
now in order to obtain benefits later. Otherwise, 
how would one accept the pain of surgery? Why 
would someone resist the temptation to have some- 
thing irresistible, or delay the gratification from 
something that is appealing? 

I propose that these complex and apparently in- 
determinist behaviors are the product of a complex 
cognitive process subserved by two separate, but in- 
teracting, neural systems that were discussed ear- 
lier: (1) an impulsive, amygdala-dependent, neural 
system for signaling the pain or pleasure of the im- 
mediate prospects of an option; and (2) a reflective, 
prefrontal-dependent, neural system for signaling 
the pain or pleasure of the future prospects of an 
option. The final decision is determined by the rela- 
tive strengths of the pain or pleasure signals associ- 
ated with immediate or future prospects. When the 
immediate prospect is unpleasant, but the future is 
more pleasant, then the positive signal of future 
prospects forms the basis for enduring the unpleas- 
antness of immediate prospects. This also occurs 
when the future prospect is even more pleasant 
than the immediate one. Otherwise, the immediate 
prospects predominate and decisions shift toward 
short-term horizons. The following is a proposal of 
how a robot should be built in order to manifest the 
properties of willpower, a key characteristic of hu- 
man decisions: 

Input of Information 

Once a registry of values linked to categories of op- 
tions or scenarios has been acquired, the robot 
should access and trigger the value assigned to each 
of these options and scenarios whenever they (or 
closely related ones) are encountered. The con- 
fronting entities and events may have many conflict- 
ing values, some of which are triggered impulsively 



through the amygdala system, and some of them 
reflectively through the prefrontal system. Another 
process modulates the strength of these values by 
factors such as time (i.e., immediate versus de- 
layed), probability (the outcome is certain or very 
probable), deprivation (e.g., hungry or not), and so 
on. These modulation effects are mediated by the 
prefrontal system, as explained by Bechara and 
Damasio (2005). 

Emotional Evaluation 

Although the input of information may trigger nu- 
merous somatic responses that conflict with each 
other, the end result is that an overall positive or 
negative somatic state (or value) emerges. We have 
proposed that the mechanisms that determine the 
nature of this overall somatic state (i.e., being pos- 
itive or negative) are consistent with the principles 
of natural selection, that is, survival of the fittest 
(Bechara & Damasio, 2005). In other words, nu- 
merous and often conflicting somatic states may 
be triggered at the same time, but stronger ones 
gain selective advantage over weaker ones. With 
each piece of information brought by cognition, 
the strength of the somatic state triggered by that 
information determines whether that same infor- 
mation is likely to be kept (i.e., brought back to 
cognition so that it triggers another somatic state 
that reinforces the previous one) or is likely to be 
discarded. Thus over the course of pondering a 
decision, positive and negative somatic markers 
that are strong are reinforced, while weak ones are 
eliminated. This process of elimination can be 
very fast. 

Decision Output 

Ultimately, a winner takes all; an overall, more 
dominant somatic state emerges (a gut feeling or a 
hunch, so to speak), which then provides signals to 
cognition that modulate activity in neural struc- 
tures involved in biasing behavioral decisions. 
Thus the more dominant an available option over 
the others, the quicker the decision output; the 
more equal and similar the available options, the 
slower the decision output. 



Conclusion 

Emotions are not, as some might feel, simply a nui- 
sance. Nor can we can ignore emotions on the 
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grounds that the greatest evolutionary development 
of the human brain is in relation to the cortex and 
its "cold" cognition, as opposed to the more primi- 
tive limbic brain, the seat of emotions. Rather, emo- 
tions are a major factor in the interaction between 
environmental conditions and human cognitive 
processes, with these emotional systems (underly- 
ing somatic state activation) providing valuable im- 
plicit or explicit signals that recruit cognitive 
processes that are most adaptive and advantageous 
for survival. Therefore, understanding the neural 
mechanisms underlying emotions, feelings, and 
their regulation is crucial for many aspects of hu- 
man behaviors and their disorders. 



Main Points 

1 . Decision making is a process guided by 
emotional signals (gut feelings) that are 
generated in anticipation of future events, 
which help bias decisions away from courses 
of action that are disadvantageous to the 
organism, or toward actions that are 
advantageous. 

2. Human factors evaluation of decision making 
in everyday and work situations must take 
these emotional signals into account. 

3. Impairments of decision making and 
inappropriate social behaviors are often 
observed after damage to neural regions that 
overlap considerably with those subserving 
the expression of emotions and the experience 
of feelings. 

4. These biasing emotional signals (gut feelings) 
do not need to be perceived consciously. 
Emotional signals may begin to bias decisions 
and guide behavior in the advantageous 
direction before conscious knowledge does. 

5. Knowledge without emotional signaling leads 
to dissociation between what one knows or 
says and how one decides to act. Patients who 
have this disconnection may manifest 
declarative knowledge of what is right and 
what is wrong, but they fail to act accordingly. 
Such patients may say the right thing, but they 
do the wrong thing. 

6. Abnormalities in emotions, feelings, decision 
making, and social behavior may be biological, 
that is, caused by damage to specific neural 
structures, but they can also be learned, such 



as acquiring or attaching the wrong value to 
actions and behaviors during childhood. 
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Stress and Neuroergonomics 



Neuroergonomics has been defined as "the study of 
human brain function in relation to work and tech- 
nology" (Parasuraman, 2003, p. 5). Given that 
stress is an aspect of many forms of work, it is 
natural that investigations of stress and its neu- 
robehavioral aspects form a core topic in neuroer- 
gonomics. We begin this chapter with a brief 
overview of the major conceptual approaches to 
the study of stress, including a short historical ac- 
count and a precis of the more recent stress theo- 
ries (e.g., Hancock & Desmond, 2001). We discuss 
how an understanding of stress helps shape and di- 
rect new, emerging concepts in neuroergonomics. 
We then consider the issue of individual differ- 
ences and the effects on stress in individuals, and 
ethical issues related to monitoring and mitigating 
stress in the workplace. 



Concepts of Stress 

In behavioral research, stress itself has traditionally 
been viewed as a source of disturbance arising from 
an individual's physical or social environment. 
However, since individuals do not react in exactly 
the same way to common conditions, it is now 
considered more appropriate to view stress in 



terms of each individual's response to his or her 
environment. This is the so-called transactional 
perspective (see Hockey, 1984, 1986; Lazarus & 
Folkman, 1984; Matthews, 2001; Wickens & Hol- 
lands, 2000). This has led to both continuing de- 
bate and some confusion, since some researchers 
continue to define stress in terms of the external 
stimuli involved (e.g., noise, temperature, vibra- 
tion; see Elliott & Eisdorfer, 1982; Jones, 1984; 
Pilcher, Nadler, & Busch, 2002). Defining stress 
only in terms of the physical stimulus does not ac- 
count for why the same stimulus induces different 
stress responses across individuals or within the 
same individual on different occasions (Hockey, 
1986; Hockey & Hamilton, 1983; Matthews, 
2001). Consideration of stimulus properties there- 
fore provides an important but nevertheless incom- 
plete understanding of stress effects. To more fully 
capture all the dimensions of stress, we have devel- 
oped a "trinity of stress" model (Hancock & Warm, 
1989), which includes (1) environmental stimula- 
tion as the input dimension; (2) the transactional 
perspective, emphasizing the individual response 
as the adaptation facet; and (3) most critically, an 
output performance level. We return to this de- 
scription after a review of some theoretical back- 
ground. 
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Arousal Theory 

In contrast to the stimulus-driven view of stress, 
which might be considered a physics-based ap- 
proach, the biologically based approaches define 
stress in terms of the physiological response pat- 
terns of the organism (Selye, 1976). In this vein, 
Cannon (1932) conceptualized stress in terms of 
autonomic nervous system activation triggered by 
a homeostatic challenge. According to Cannon, ex- 
ternal conditions (e.g., temperature, noise, etc.) or 
internal deficiencies (e.g., low blood sugar, etc.) 
trigger deviations from homeostatic equilibrium. 
Such threats to equilibrium result in physiological 
responses aimed at countering that threat. These 
responses involve sympathetic activation of the 
adrenal medulla and the release of several hormones 
(see Asterita, 1985; Frankenhaeuser, 1986; for an 
early review see Dunbar, 1954). The response-based 
approach of Cannon was also championed by Selye 
(1976), who defined stress in terms of an orches- 
trated set of these bodily defense reactions against 
any form of noxious stimulation. This set of physi- 
ological reactions and processes he referred to as 
the general adaptation syndrome. Environmental ob- 
jects or events that give rise to such responses were 
referred to as stressors. Within Selye 's theory, physi- 
ological responses to stressors are general in char- 
acter, since the set of responses is similar across 
different stressors and contexts. 

Arousal theory is one of the most widely ap- 
plied physiological response-based theories of 
stress and performance (Hebb, 1955). Arousal level 
is a hypothetical construct representing a nonspe- 
cific (general) indicator of the level of stimulation of 
the organism as a whole (Hockey, 1984). Arousal 
may be assessed using techniques such as elec- 
troencephalography (EEG) or indicators of auto- 
nomic nervous system activity such as the galvanic 
skin response (GSR) and heart rate. As a person be- 
comes more aroused, the waveforms of the EEG in- 
crease in frequency and decrease in amplitude (see 
also chapter 2, this volume), and skin conductance 
and heart rate increase (see also chapter 14, this 
volume). 

Within this framework, stress effects are ob- 
served under conditions that either overarouse (e.g., 
noise) or underarouse the individual (e.g., sleep 
deprivation; Hockey, 1984; McGrath, 1970). This 
approach assumes an inverted-U relationship be- 
tween arousal and performance — the Yekes-Dodson 



law — such that the optimal level of performance 
is observed for midrange levels of arousal (Hebb, 
1955). Stressors, such as noise or sleep loss, act by 
either increasing or decreasing the arousal level of 
the individual relative to the optimum level for a 
given task (Hockey & Hamilton, 1983). The opti- 
mum level is also postulated to be inversely related 
to the difficulty of the task (Hockey, 1984, 1986). 
A potential mechanism to account for this relation 
was first postulated by Easterbrook (1959), who 
indicated that emotional arousal restricts the uti- 
lization of the range of peripheral visual cues from 
the sensory environment, so that, under conditions 
of chronic or acute stress, peripheral stimuli are 
less likely to be processed than more centrally lo- 
cated cues. As attention narrows (i.e., as the num- 
ber of cues attended to is reduced), performance 
capacity is preserved by the retention of focus on 
salient cues. Eventually, performance fails as stress 
increases and even salient cues become excluded. 
Hancock and Dirkin (1983) showed that the nar- 
rowing phenomenon was attentional rather than 
sensory in nature, since individuals narrowed to 
the source of greatest perceived salience wherever 
it appeared in the visual field. More recently, 
Hancock and Weaver (2005) showed that the nar- 
rowing phenomenon demonstrated in spatial per- 
ception is also evident in the temporal domain, as 
in the eifects of stress on time perception (see also 
Szalma &r Hancock, 2002). 

There are several problems with the traditional 
arousal explanation of stress and performance. 
First, the different physiological indices of arousal 
often do not correlate well. In the performance of 
a vigilance task, for instance, muscle tension, as 
measured by electromyogram, and catecholamine 
levels can indicate a highly aroused state, but skin 
conductance might indicate that the observer is de- 
aroused (Hovanitz, Chin, & Warm, 1989; Parasur- 
aman, 1984). Second, it has proven difficult to 
define the effects of stressors on arousal indepen- 
dent of effects on performance (Hockey, 1986). 
Third, the theory can accommodate almost any re- 
sults, making it a post hoc explanation that is diffi- 
cult to falsiiy (i.e., test empirically; Hancock & 
Ganey, 2003; Hockey, 1984; Holland & Hancock, 
1991). Finally, arousal theory assumes that a stres- 
sor (or set of stressors) affects overall processing ef- 
ficiency and that differences in task demands (i.e., 
difficulty) are reflected only in the position of the 
optimal level of performance. Hockey and Hamilton 
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(1983) noted, however, that environmental stres- 
sors can have differential effects on the pattern of 
cognitive activity, and a single dimension, as posited 
by arousal theory, cannot account for such differ- 
ences among stressors. Hence, a multidimensional 
approach is necessary in order to understand stress 
effects, with arousal mechanisms representing only 
one facet of this complex construct. 



Appraisal and Regulatory Theories 

Most modern theories of stress and performance 
have two central themes. They either explicitly in- 
clude or implicitly assume an appraisal mechanism 
by which individuals assess their environments 
and select coping strategies to deal with those envi- 
ronments (see Hancock & Warm, 1989; Hockey, 
1997; Lazarus & Folkman, 1984). Indeed, Lazarus 
and Folkman defined psychological stress itself as 
the result of an individual's appraisal of his or her 
environment as being taxing or exceeding his or 
her resources or endangering his or her well-being. 
The negative effects of stress are most likely to oc- 
cur when individuals view an event as a threat (pri- 
mary appraisal) and when they assess their coping 
skills as inadequate for handling the stressor (sec- 
ondary appraisal; see Lazarus & Folkman, 1984). 
Both the person-environment interactions and the 
appraisal processes are likely organized at multiple 
levels (Matthews, 2001; Teasdale, 1999). 

A second central theme of current stress theo- 
ries is that individuals regulate their internal states 
and adapt to perturbations resulting from external 
stressors, including social stressors and task-based 
stress. Individuals respond to appraised threats 
(including task load) by exerting compensatory 
effort to either regulate their internal cognitive- 
affective state or to recruit resources necessary to 
maintain task performance. Thus, individuals are 
often able to maintain performance levels, particu- 
larly in real-world settings, but only at a psycho- 
logical and physiological cost (Hancock & Warm, 
1989; Hockey, 1997). Two current models of stress 
and performance that emphasize these regulatory 
mechanisms and adaptation are those of Hockey 
(1997) and Hancock and Warm (1989). 

Hockeys (1997) theory is based on assump- 
tions that behavior is goal directed and controlled 
by self-regulatory processes that have energetic 
costs associated with them. He distinguished be- 



tween effort as controlled processing (e.g., working 
memory capacity) and effort as compensatory con- 
trol (e.g., arousal level). Hockey proposed that 
mental resources and effort are allocated and con- 
trolled via two levels of negative feedback regula- 
tion by an effort monitor that compares current 
activity to goal-based performance standards. Sim- 
ple, well-learned tasks are controlled at a lower 
level that requires very little effort (i.e., very low re- 
source allocation) to maintain performance goals. 
When demands are placed on the cognitive system, 
via increased task demand or other forms of stress, 
control is exerted by a supervisory controller at the 
higher level. At this level, resources are recruited to 
compensate for goal discrepancies created by the 
increased demands, and information processing is 
more controlled and effortful. Note, however, that 
this represents only one potential response to 
stress. A second possibility would be to alter the 
task goals to maintain low effort or to reduce effort 
in the face of prolonged exposure to stress. The ef- 
fortful coping response was referred to as a strain 
mode, and the reduction of effort or performance 
goals is passive coping mode. Hockey's model pro- 
vides a flexible structure and an energetic mecha- 
nism by which the effects that environmental 
demands (i.e., stress) place on the cognitive system 
can be understood. As we shall see, however, this 
view rests on concepts that are difficult to test 
empirically. Specifically, the "resource concept" em- 
ployed in Hockey's model, which came to domi- 
nate stress theory after the fall of the unitary 
arousal explanation, presents a problem in that it 
is ill defined and difficult to quantify. It is our 
contention that neuroergonomics has a significant 
contribution to make in improving the resource 
concept, since such "mental energy" must be specif- 
ically understood if neuroergonomic methods are to 
be effective. 

The approach presented by Hancock and 
Warm (1989) also adopts as a fundamental tenet 
the idea that in many stressful situations humans 
adapt to their environments. This adaptation is 
manifested in an extended inverted-U function that 
describes performance change as a function of stress 
level, as shown in figure 13.1. Stress here can take 
the form of both overstimulation (hyperstress), in 
which the sensory systems experience an elevated 
level of stimulation, and understimulation (hy- 
postress), in which the sensory systems receive 
disturbingly little stimulation. Note that for a wide 
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Figure 1 3.1 . The stress-adaptation model of Hancock and Warm (1989). 
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range of stimulation or task demand, individuals 
maintain behavioral and physiological stability. 
There are multiple levels of adaptation that are each 
nested within the general, extended-U function. 
Thus subjective state (e.g., the normative and com- 
fort zones in figure 13.1) is altered by relatively mi- 
nor levels of disturbance, whereas it takes a greater 
degree of disturbance to affect behavioral perfor- 
mance, which is itself less robust than physiological 
response capacity. These different facets of response 
are linked: Incipient failure at one level represents 
the beginning of stress disturbance at the next level. 
While each level retains the same extended-U 
shape, the nesting represents the progressive fragility 
across levels. In the same way, there are comparable 
divisions within levels. For example, within the 
physiological level, there are progressive functions 
for cell, organelle, organ, and so on. 

Hancock and Warm's model explicitly recog- 
nizes that tasks themselves often represent the 
proximal and most salient source of stress. Thus, 
the adaptation level of an individual will heavily de- 
pend upon the characteristics of the tasks facing 
that individual. Hancock and Warm (1989) pro- 
posed two fundamental dimensions along which 
tasks vary, these being information rate (the speed 
with which demands are made) and information 
structure (the complexity of that demand). Com- 
bined variations in task and environmental demand 



impose considerable stress on operators, to which 
they adapt via various coping efforts. Breakdown of 
performance under stress, and its inverse, efficient 
behavioral adaptability, occurs at both psychologi- 
cal and physiological levels, with psychological 
adaptability failing before comparable physiological 
adaptability (for a related view, see Matthews, 2001). 
A representation of the adaptive space in the con- 
text of the extended U is shown in figure 13.2. To 
locate an individual's level of adaptation to a set of 
stressors in an environment, one defines vectors for 
the level of stress and cognitive and physiological 
state, as well as the position of task performance 
along the space-time axes. Although further work is 
required to quantify the theoretical propositions in 
the Hancock and Warm (1989) model, we argue 
that such quantification will result in a rubric under 
which neuroergonomic measures of stress can be 
developed in coordination with performance and 
subjective measures. For instance, if subjective state 
or comfort declines prior to task performance, this 
should be observable not only via self-report but 
also using well-defined neural measures with well- 
validated links to cognitive processes. If the task di- 
mensions can be specified precisely, predictions can 
be made regarding the level of adaptation under dif- 
ferent task and arousal conditions. Neuroergonomic 
methods can thereby facilitate tests of theoretical 
models of stress such as that discussed here. 
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Figure 1 3.2. The stress-adaptation model of Hancock and Warm (1989), showing the underlying task dimensions 
that influence behavioral and physiological response to stress. Letters A, B, C, and D represent boundaries (toler- 
ance limits) for physiological function (A), behavior/performance (B), subjective comfort (C), and the normative 
zone (D). The vector diagram at upper right illustrates how the multiple variables (i.e., information rate, informa- 
tion structure, and stress level) can be combined into a single vector representing cognitive and physiological state. 



Stress and Neuroergonomics Research 

The above considerations show that stress, like 
many psychological constructs, is difficult to define 
precisely. This represents a challenge for neuroer- 
gonomics because to identify a particular neurolog- 
ical state as behaviorally stressful requires a 
well-defined stress concept. It is also an opportu- 
nity because consideration of specific neural mech- 
anisms for stress responses can serve to inform 
neuroergonomic actions at the most critical opera- 
tional times. Indeed, if neuroergonomics fulfills its 
potential (see Hancock & Szalma, 2003; Parasura- 
man, 2003), it may transform the concept of stress 
itself. If, ultimately, cognitive states can be strongly 
tied to specific neurological processes, then compo- 
nents of stress that are currently defined psycholog- 
ically (e.g., appraisal, coping, worry, distress, task 
engagement, etc.) may be defined in terms of their 
underlying neural structure and function. A note of 
caution is in order, however, since this reductionis- 
tic aspiration is unlikely to be fulfilled completely. 



While the mechanisms by which appraisals occur 
are likely to be universal and nomothetic, there are 
almost inevitably individual differences in how 
thorough an appraisal is, how long it takes, and 
what aspects of the environment are attended to 
(Scherer, 1999). These may not have common neu- 
rological mechanisms. Thus, there are individual 
differences that occur spatially (what part of the en- 
vironment has drawn one's attention, and what is its 
relevance to the individual) and temporally (what is 
the time course of appraisal, how long does it take, 
and is more time spent appraising some criteria 
over others?). Neurological activity associated with 
appraisal mechanisms will very probably not be 
identical across these two dimensions. For a suc- 
cessful neuroergonomic approach to stress, we will 
need to have the capacity to examine the neural cor- 
relates of specific cognitive patterns (cf. Hockey & 
Hamilton, 1983) and distinguish among varieties of 
appraisal and coping mechanisms. 

The multidimensionality of stress (Matthews, 
2001) and the likely hierarchical organization 
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of appraisal mechanisms (Teasdale, 1999) imply 
that neuroergonomic measures will have to be 
sufficiently sensitive to delineate these dimensions 
of stress. For instance, considering the state "Big 3" 
(worry, distress, and task engagement; see Matthews 
et al., 1999, 2002), neuroergonomic tools would 
need to have the capacity to differentiate the neural 
processes underlying worry (cognitive), distress 
(mood, cognitive), and task engagement (mood, en- 
ergetic, cognitive). Further, each of these dimen- 
sions has its specific components. For instance, task 
engagement consists of motivation, concentration, 
and energetic arousal (see Matthews et al., 2002). 
For a neuroergonomic system to adapt to operator 
stress, it will need to be sensitive to such facets of 
operator state. In addition, a useful neuroergonomic 
study of stress would be to develop valid neurologi- 
cal indices of the progressive "shoulders of failure" 
depicted in the Hancock and Warm (1989) model 
(see figure 13.1). For instance, indices that can track 
and predict the conditions under which an individ- 
ual will transition from one curve to the next (e.g., 
from the comfort zone to failure of behavioral adap- 
tation) would be very useful in aiding operator 
adaptation to stressful environments. 

A core problem in development of stress theory 
is in the use of resource theory (Wickens, 1984) as 
an explanatory framework. With the fall of the uni- 
tary arousal concept, resource models emerged as a 
primary intervening variable to account for perfor- 
mance effects (see Hockey, Gaillard, & Coles, 1986). 
While resource theory has found some support and 
has been used in theoretical models of stress (e.g., 
Gopher, 1986; Hockey, 1997), it has been criticized 
as inadequate to the task of explaining attention al- 
location and human performance (e.g., Navon, 
1984). In addition, the structure of resources and 
the mechanisms of resource control and allocation 
may not be common across individuals (Thropp, 
Szalma, & Hancock, 2004). The problem for stress 
theory is that a vague construct (resources) was em- 
ployed to explain mechanisms by which another 
vague construct (stress) impacts information pro- 
cessing and performance. A significant contribution 
of neuroergonomics lies in its demand for precision, 
which only then permits computer and engineering- 
mediated changes in the effects of stress on informa- 
tion processing and performance. However, one 
must avoid the temptation to overly simplistic re- 
ductionism and recognize that neuroscience can en- 
hance our understanding of resources (as both 



energetic states and processing capacity) but not 
necessarily replace psychological constructs with 
purely neurophysiological mechanisms. Fundamen- 
tally, the question for the neuroergonomics ap- 
proach to stress is the same as for other approaches: 
How is it that individuals generally adapt to stress 
and maintain performance and what are the cogni- 
tive and perceptual mechanisms by which this oc- 
curs? The challenge for neuroergonomic efforts to 
answer this question will be development of neuro- 
logical measures that are not merely outputs of in- 
formation processing but reflect the processing itself 
and provide a direct measure of brain state. 

To the degree that neuroergonomics can illu- 
minate the above issues, it will also facilitate the 
development of more precise theory regarding the 
associations, dissociations, and insensitivities be- 
tween performance and workload (Hancock, 1996; 
Oron-Gilad, Szalma, Stafford, & Hancock, 2005; 
Parasuraman & Hancock, 2001; Yeh & Wickens, 
1988), as well as distinguishing between effortful 
and relatively effortless performance (cf. processing 
efficiency theory; see Eysenck & Calvo, 1992; and 
see also Hockey, 1997). Eventually, neuroer- 
gonomic approaches to stress research can lead to 
improving not only physiological measurement of 
workload but also in relating such measures to 
other forms of workload measurement (i.e., perfor- 
mance and subjective measures; see O'Donnell & 
Eggemeier, 1986). 



Validation of Neuroergonomic 
Stress Measures 

A fundamental problem facing those pursuing re- 
search in neuroergonomics is measurement (Han- 
cock & Szalma, 2003). How does one connect 
mental processes to overt behavior (or physiologi- 
cal outputs) in a valid and reliable fashion? Driving 
this development of sound measures is the neces- 
sity for sound theory. If one adopts a monistic, re- 
ductionistic approach, then the ultimate result for 
neuroscience and neuroergonomics is the attempt 
to replace psychological models of stress with neu- 
rological models that specify the brain mechanisms 
that produce particular classes of appraisal and 
coping responses. 

The alternative position that we posit here is a 
functionalist approach (see also Parasuraman, 2003) 
in which one postulates distinct physical and psy- 
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etiological constructs for a complete understanding 
of stress and cognition. From this perspective, neu- 
roscience provides another vista into understanding 
stress and cognition that complements psychologi- 
cal evidence. Whichever position is taken, one must 
still base neuroergonomic principles on sound the- 
oretical models of psychological and neurological 
function. As we have indicated in our earlier work, 
understanding multidimensional concepts, includ- 
ing stress, requires a multimethod assessment so 
that a more complete picture of cognition and cog- 
nitive state can be revealed (Oron-Gilad et al, 
2005). 



Commonalities and Differences 
between Individuals 

Neuroergonomics is a logical extension of the criti- 
cal need for stress researchers to consider individual 
variation in stress response. Because stress response 
varies as a function of task demand and the physi- 
cal, social, and organizational context, it is also 
likely that neuroergonomic stress profiles will vary 
between and within individuals. Indeed, it is already 
known that individuals vary in cortical arousal and 
limbic activation and that indices of these covary 
with personality traits (Eysenck, 1967; Eysenck & 
Eysenck, 1985), although the evidence for a causal 
link is mixed (e.g., see Matthews & Amelang, 1993; 
for a review, see Matthews, Deary, & Whiteman, 
2003). Application of emerging neuroergonomic 
technologies to adaptive systems will provide in- 
formation, combined with performance differences 
associated with specific traits and states, that can 
be used to adjust systems to particular operators 
and to adapt as operator states change over time. 
Further, application of neuroscience techniques 
might enhance our theoretical understating of indi- 
vidual differences in stress response. Although 
many theories of stress have been developed, com- 
prehensive theory on individual differences in 
stress and coping in ergonomic domains has been 
lacking. Neuroergonomics offers a new set of tools 
for individual differences researchers for both em- 
pirical investigation and theory development. 

Individual differences in cognition, personality, 
and other aspects of behavior have traditionally 
been examined using the psychometric approach. 
Understanding the sources of such differences has 
also been illuminated by behavioral genetic studies 



of psychometric test performance. For example, this 
method has been used to show that general intelli- 
gence, or g, is highly heritable (Plomin & Crabbe, 
2000). However, conventional behavioral genetics 
cannot identify the particular genes involved in in- 
telligence or personality. The spectacular advances 
in molecular genetics now allow a complementary 
approach to behavioral genetics — allelic association. 
In this approach, normal variations in single genes, 
identified using DNA genotyping, are associated 
with individual differences in performance on cog- 
nitive tests. This method has been applied to the 
study of individual differences in cognition in 
healthy individuals, revealing evidence of modula- 
tion of attention and working memory by specific 
genes (Parasuraman, Greenwood, Kumar, & Fos- 
sella, 2005). Parasuraman and Caggiano (2005) 
have incorporated this approach into their neuroer- 
gonomic framework and discussed how molecular 
genetics can pinpoint the sources of individual dif- 
ferences and thereby provide new insights into tra- 
ditional issues of selection and training. This 
approach could also be applied to examining indi- 
vidual differences in stress response. 



Hedonomics and Positive Psychology 

Thus far we have been concerned primarily with 
implications of neuroergonomics for stress. There 
is also an opportunity to explore the application 
of neuroergonomics to the antithesis of stressful 
conditions. The latter study has been termed hedo- 
nomics, which is defined as "that branch of sci- 
ence which facilitates the pleasant or enjoyable 
aspects of human-technology interaction" (Han- 
cock, Pepe, & Murphy, 2005, p. 8). This repre- 
sents an effort not simply to alleviate the bad but 
also to promote the good. 

Studies of the level of pleasure experienced by 
individuals are rooted in classic effects of limbic 
stimulation on behavior (Olds & Milner, 1954). In- 
deed, neuroergonomic indices of attention might 
clarify the processes that occur for maladaptive ver- 
sus adaptive attentional narrowing, recovery from 
stress and performance degradation (i.e., hystere- 
sis). In addition, neuroergonomics can contribute 
to the emerging positive psychology trend (Selig- 
man & Csikszentmihalyi, 2000) by identifying the 
neurological processes underlying flow states (Csik- 
szentmihalyi, 1990) in which individuals are fully 
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engaged in a task and information processing is 
more automatic and fluid rather than controlled 
and effortful. 



Ethical Issues in Neuroergonomics 
and Stress 

There is a pervasive question that must attend the 
development of all new human-machine technolo- 
gies, and that is the issue of purpose (Hancock, 
1997). Subsumed under purpose are issues such 
as morality, benefit, ethics, aesthetics, cost, and 
the like. As we have merely introduced elsewhere 
(Hancock & Szalma, 2003), there are issues of pri- 
vacy, information ownership, and freedom that will 
also have to be considered as cognitive neuro- 
science develops and is applied to the design of 
work. In regard to stress, there is a particular dan- 
ger that those who are naturally prone to specific 
patterns of stress response (e.g., those high in neu- 
roticism or trait anxiety) may be excluded from 
opportunities or even punished by controlling 
authorities for their personality characteristics. If 
we assume, however, that the rights of individuals 
have been secured, application of neuroergonomic 
technologies to monitor emotional states of indi- 
viduals could provide useful information for ad- 
justing system status to the characteristics of the 
individual. Thus, those who are prone to trait anxi- 
ety could be assisted by an automated system when 
it is detected that their state anxiety is increasing to 
levels that make errors or other performance fail- 
ures more likely. In such an application, the tech- 
nology could serve to increase the performance of 
anxious individuals to a level comparable to that of 
individuals low in trait anxiety. More generally, 
neuroergonomics offers a new avenue for research 
and application in individual differences in coping 
and responses to stress, as a technology of inclu- 
sion rather than a technology of exclusion. In 
many modern systems, the environment is suffi- 
ciently flexible that it can be adapted to the charac- 
teristics and current cognitive state of the individual 
operator. This can improve performance, reduce 
the workload and stress associated with the task, 
and perhaps even render the task more enjoyable. 

As neuroergonomic interventions emerge, 
consideration of intention and privacy will be cru- 
cial. In the age of ever-increasing technology, the 
opportunity for any individual to preserve his or 



her own private experience is vastly diminished. 
The proliferation of video technology alone means 
that events that have previously remained hidden 
behind a barrier of institutional silence now be- 
come open to public inspection. The modern threat 
of terrorism has also engendered new categories of 
surveillance technologies which use sophisticated 
software that seeks to process the nominal intent of 
individuals. Added to these developments are the 
evolutions in detection technologies that use evi- 
dence such as DNA traces to track the presence of 
specific individuals. Such developments indicate 
that overt actions are now detectable and record- 
able and thus potentially punishable. Further, this 
invasion is moving to the realm of speech and com- 
munication. Recorded by sophisticated technolo- 
gies, utterances regarding (allegedly) maleficent 
intent now become culpable evidence. Despite a 
nominal freedom of expression, one cannot offer 
threats, conspire to harm, or engage in certain 
forms of conversation without the threat of detec- 
tion and punishment. Many would argue that de- 
structive acts should be punished, and that voicing 
violent intent is also culpable, whether the intent 
is fulfilled or not (hence the ambivalence regard- 
ing jokes concerning exploding devices at airport 
screening facilities). However, neuroergonomics, 
if the vision is fully or even partly fulfilled, now 
promises to extend these trends further. It will not 
only be actions and language that could be consid- 
ered culpable but, more dangerously, the thought 
itself. As Marcus Aurelius rightly noted, we are the 
sole observer and arbiter of our own personal pri- 
vate experience. We may safely and unimpeachably 
think unpleasant thoughts about anyone or any- 
thing, content in the knowledge that our thoughts 
are private and impenetrable to any agency or indi- 
vidual. Neuroergonomics could threaten this fun- 
damental right and indeed, in so doing, threaten 
what it is to be an individual human being. 

We paint this picture not to discourage the pur- 
suit of neuroergonomics, but rather to sound a cau- 
tionary alarm. While neuroergonomic interventions 
have the potential to provide early warning that an 
individual is overstressed and therefore could be 
used to mitigate the negative effects of such stress, 
the imposition of the neuroergonomic intervention 
could itself impose significant stress on the individ- 
ual, particularly if those in authority (e.g., company 
management, governmental agencies) have access 
to the information provided by neurological and 
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physiological measures, and individuals appraise 
this situation as a potential threat with which they 
cannot effectively cope. 



Conclusion 

Clearly, one application of neuroergonomics to 
stress will be a set of additional tools to monitor op- 
erator state for markers predictive of later perfor- 
mance failure and, based on such information, 
adjusting system behavior to mitigate the stress. 
However, to reduce stress effectively requires an un- 
derstanding of its etiology and the mechanisms by 
which it occurs and affects performance. To address 
these issues requires development of good theoreti- 
cal models, and we see the neuroergonomic ap- 
proach to stress research making a significant 
contribution toward this development. As a multi- 
dimensional construct, stress requires multidimen- 
sional assessment, and improvement of theory 
requires that the relations among the dimensions be 
well articulated. Neuroergonomics will facilitate 
this process. The key will be to develop reliable and 
valid neurological metrics relating to cognitive and 
energetic states and the processes underlying the re- 
cruitment, allocation, and depletion of cognitive re- 
sources. Our optimism is not unguarded, however. 
First, neurological measures should be viewed as 
one piece of a complex puzzle rather than as a re- 
ductive replacement for other indices of stress. 
Omission of psychological constructs and measures 
would weaken the positive impact of neuroscience 
for stress research and stress mitigation efforts. Sec- 
ond, in implementing stress research, we must en- 
sure that the application of neuroergonomics to 
stress mitigation does not increase stress by impos- 
ing violations of security and privacy for the indi- 
vidual. If these concerns are adequately addressed, 
neuroergonomics could not only revolutionize 
stress as a psychological construct but could also 
serve to transform the experience of stress itself. 

On a more general front, neuroergonomics 
could be used to mitigate stress completely. With 
sufficient understanding of neural stress states and 
their precursors in both the environment and the 
operator's appraisal of that environment, it would 
be possible, in theory, to develop a servomechanis- 
tic system whereby sources of stress were ablated as 
soon as they arose. But would this be a good thing? 
It is not that we enjoy certain adverse situations, 



but it may be that it is the stimulation of such ad- 
verse conditions that spur us to higher achieve- 
ment. This brings us to our final observations on 
the very thorny issue of the purpose of technology 
itself. Is the purpose of technology to eradicate all 
human need and as a corollary to this, to instantly 
and effortlessly grant all human physical desires? 
We suggest not. Indeed, such a state of apparent 
dolcejar niente (life without care) might well prove 
the equivalent of the medieval view of hell! Further, 
it is currently unclear how to respond to mistakes 
in neuroergonomics if the purpose is to facilitate the 
immediate transition from an intention to an action. 
Complex error recovery processes are built into the 
current human motor system — will such effective 
guards be embedded into future neuroergonomics 
systems? Some scientists have opined that all tech- 
nology is value neutral in that it can be employed 
for both laudable and detestable purposes. But this 
is a flawed argument because the creation of each 
new embodiment of technology is an intentional act 
that itself expresses an intrinsic value-based deci- 
sion (see Hancock, 1997). To whatever degree that 
value is apparently hidden in the process of concep- 
tion, design, and fabrication, and to whatever de- 
gree others choose to pervert that original intention, 
the act itself implies value. Thus, we need at the 
present stage of development to consider not 
whether we can develop neuroergonomics tech- 
nologies but rather, whether we should. Needless to 
say, this is liable to be a somewhat futile discussion 
since rarely, if ever, in human history have we re- 
frained from doing what is conceived as being pos- 
sible, whether we should or not. Indeed, it is this 
very motivation that may well be the demise of the 
species. To end on a more hopeful note: perhaps 
not. Perhaps the present capitalist-driven global 
structure will consider the greater good of all indi- 
viduals (and indeed all life) and refrain from the 
crass, materialistic exploitation of whatever inno- 
vations are realized. Then again — perhaps not. And 
that was a hopeful note. 



Main Points 

1 . Stress is a multidimensional construct that 
requires multidimensional assessment. 
Neuroergonomics promises to provide 
valuable tools for this effort. 
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2. A major problem for stress research is the 
difficulty in precisely defining the concepts of 
stress and cognitive resources. 
Neuroergonomic efforts should be directed 
toward elucidating these constructs. 

3. Neuroergonomics can improve the state of 
stress theory via programmatic research 
toward establishing the links between 
neurological and cognitive states. This is 
critical, since the validity of neuroergonomic 
measures depends heavily on sound 
psychological theory. 

4. A potential practical application for stress 
mitigation will be the ability to monitor 
operator state in real time so that systems can 
adapt to those states as operators experience 
stress. Such efforts, in the context of 
neuroergonomics, are already underway. 

5. Neuroergonomics can also be useful for 
studying individual differences in stress and 
coping and establishing general theoretical 
framework for individual differences in perfor- 
mance, stress, and workload. 

6. While the promise for neuroergonomics is 
high, we must ensure that individuals' privacy 
and well-being are preserved so that the cure 
does not become worse than the disease. 
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Sleep and Circadian Control 
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Overview of Sleep and 
Circadian Rhythms 

Most organisms show daily changes in their behav- 
ior and physiology that are not simply controlled 
by external stimuli in the environment. In mam- 
mals, these 24-hour cycles, otherwise knows as cir- 
cadian rhythms, are primarily controlled by an 
internal clock called the suprachiasmatic nucleus 
(SCN), located in the hypothalamus (Moore, 
1999). These cycles can be synchronized to exter- 
nal time signals, but they can also persist in the ab- 
sence of such signals. In the absence of time cues, 
in humans, the SCN shows an average "free run- 
ning" intrinsic period of 24.18 hours (Czeisler et 
al, 1999). However, the SCN is entrained to the 
24-hour day via zeitgebers, or time givers, of which 
the strongest is light. This endogenous circadian 
pacemaker affects many physiological functions, 
including core body temperature, plasma Cortisol, 
plasma melatonin, alertness, and sleep patterns. 
The nadir of the circadian component of the en- 
dogenous core body temperature rhythm is associ- 
ated with an increased sleep propensity (Dijk & 
Czeisler, 1995). 

For most animals, the timing of sleep and wake- 
fulness under natural conditions is in synchrony 



with the circadian control of the sleep cycle and all 
other circadian-controlled rhythms. Humans, how- 
ever, have the unique ability to cognitively override 
their internal biological clock and its rhythmic out- 
puts. When the sleep-wake cycle is out of phase 
with the endogenous rhythms that are controlled by 
the circadian clock (e.g., during night shift work or 
rapid travel across time zones), adverse effects will 
result (Dijk & Czeisler, 1995). The synchrony of an 
organism with both its external and internal envi- 
ronments is critical to the organisms well-being and 
survival. A disruption of this synchrony can result 
in a range of difficulties, such as impaired cognitive 
function, sleepiness, altered hormonal function, and 
gastrointestinal complaints. 

In addition to the circadian component, an- 
other fundamental regulatory process is involved in 
programming sleep and alertness. The less (good- 
quality) sleep one obtains daily, the greater will be 
the homeostatic drive for sleep. The neurobiological 
mechanisms underlying homeostatic sleep pressure 
are beginning to be identified (Mignot, Taheri, & 
Nishino, 2002; Saper, Chou, & Scammell, 2001). 
The mechanisms promoting wakefulness and sleep 
appear to involve a number of neurotransmitters 
and nuclei located in the forebrain, midbrain, and 
brain stem (Mignot, Taheri, & Nishino, 2002; Saper, 
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Chou, & Scammell , 2001). For example, one the- 
ory posits that adenosine tracks lost sleep and may 
induce sleep when the homeostat is elevated due 
to being awake for too long a period, or when 
sleep quality or quantity are chronically inadequate 
(Basheer, Strecker, Thakkar, & McCarley 2004). 
The homeostatic regulation of sleep interacts non- 
linearly with the circadian cycle to produce dy- 
namic changes in the propensity and stability of 
sleep and waking across each 24-hour period and 
across days (Van Dongen & Dinges, 2000). 

Slow-wave sleep — and especially slow-wave 
brain activity during sleep — are currently consid- 
ered the classical markers of the sleep homeostatic 
process. There is evidence that the time course of 
the sleep homeostatic process can also be monitored 
during wakefulness in humans using oculomo- 
tor measures (see below). In addition, electroen- 
cephalographic (EEG) studies have suggested that 
certain slow-frequency components of brain activity 
increase as the duration of wakefulness increases. 
Scheduling multiple naps during the day attenuates 
this increase in activity during wakefulness (Ca- 
jochen, Knoblauch, Krauchi, Renz, & Wirz-Justice, 
2001). It thus appears that low-frequency compo- 
nents of EEG during wakefulness are closely associ- 
ated with sleep homeostasis. Together, homeostatic 
and circadian processes determine degree and tim- 
ing of daytime alertness and cognitive performance 
(Czeisler & Khalsa, 2000; Durmer & Dinges, 2005; 
Van Dongen & Dinges, 2000). 



Types of Sleep Deprivation 

Sleep deprivation can result from either partial or 
total loss of sleep, which can be either voluntary 
or involuntary, and it can range in duration from 
acute to chronic. Partial sleep deprivation occurs 
when an individual is prevented from obtaining a 
portion of the sleep needed to produce normal 
waking alertness during the daytime. Typically, 
this occurs when an individual's sleep time is re- 
stricted in duration or is fragmented for environ- 
mental or medical reasons. The effects of both 
acute and chronic partial sleep deprivation on a 
range of neurobehavioral and physiological vari- 
ables have been examined over the years using a 
variety of protocols. These have included restrict- 
ing time in bed for sleep opportunities in continu- 
ous and distributed schedules, gradual reductions 



in sleep duration over time, selective deprivation 
of specific sleep stages, and situations where the 
time in bed is reduced to a percentage of the indi- 
vidual's habitual time in bed. 

Many early experiments on the effects of 
chronic partial sleep restriction concluded that vol- 
untarily reducing nightly sleep duration to be- 
tween 4 and 6 hours had little adverse effect. This 
led to the belief that one could adapt to reduced 
sleep. However, most of these early studies lacked 
key experimental controls and relied on small sam- 
ple sizes and limited measurements (Dinges, Bay- 
nard, & Rogers, 2005). More recent controlled 
experiments on the cognitive and neurobehavioral 
effects of chronic partial sleep restriction corrected 
for these methodological weaknesses and found 
that the loss of alertness and performance capabil- 
ity from chronic sleep restriction got worse across 
days and showed dose-response relationships to 
the amount of sleep obtained (Belenky et al, 2003; 
Dinges et al., 1997; Van Dongen, Maislin, Mulling- 
ton, & Dinges, 2003). When sleep duration was 
reduced below 7 hours per night, daytime func- 
tions deteriorated. 

In contrast to partial sleep deprivation, total 
sleep deprivation occurs when no sleep is obtained 
and the waking period exceeds 16 hours in a 
healthy adult. Total sleep deprivation that extends 
beyond 24 hours — such as occurs in sustained or 
critical operations — reveals the nonlinear interac- 
tion of the escalating sleep homeostat and the en- 
dogenous circadian clock (Van Dongen & Dinges, 
2000). This interaction manifests a counterintu- 
itive outcome — namely, an individual who remains 
awake for 40 hours (i.e., a day, a night, and a sec- 
ond day) is less impaired from sleepiness at 36-38 
hours of wakefulness than at 22-24 hours of wake- 
fulness. Figure 14.1 displays this interaction mani- 
festing in a number of cognitive functions during 
acute total sleep deprivation. 

There is a third condition that can interact with 
sleep deprivation and endogenous circadian phase, 
called sleep inertia. Sleep inertia describes the grog- 
giness and disorientation that a person feels for 
minutes to hours after awakening from sleep. It is a 
transitional brain state between sleep and waking, 
which can have severe effects on cognitive functions 
involving attention and memory. Sleep inertia inter- 
acts with sleep homeostatic drive and circadian 
phase. If prior sleep duration is normal (e.g., more 
than 7 hours), sleep inertia is typically modest and 
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Figure 14.1. Performance decrements on five cogni- 
tive tasks during 88 hours of continual wakefulness. In 
this experiment, subjects (N = 24) were tested every 
2 hours on a 30-minute performance battery, beginning 
at 8 a.m. on the first day. The panels show performance 



short-lived (Achermann, Werth, Dijk, & Borbely, 
1995). The sleep stage prior to awakening is also an 
important factor in amplifying sleep inertia in that 
waking up from slow-wave sleep (SWS) is worse 
than waking up from REM (rapid eye movement) 
sleep (Ferrara, De Gennaro, & Bertini, 1999). Addi- 
tionally, it has been found that the existence of prior 
sleep deprivation increases the intensity and dura- 
tion of sleep inertia, as does awakening from sleep 
near the circadian nadir (Naitoh, Kelly, & Babkoff, 
1993). Interestingly, there is evidence that caffeine 
can block sleep inertia (Doran, Van Dongen, & 
Dinges, 2001), which may explain why this com- 
mon stimulant is much sought after in the morning, 
after a night of sleep. 



Neurobehavioral and Neurocognitive 
Consequences of Inadequate Sleep 

Cognitive performance degrades with sleep loss, 
which in operational environments is often re- 
ferred to as the effect of fatigue — however, this 
latter term has historically also referred to perfor- 
mance decrements as a function of time on task 
(i.e., work time). There is now extensive evidence 
to support the view that a vast majority of in- 
stances in which fatigue affects performance are 
due directly to inadequate sleep or to functioning 
at a nonoptimal circadian phase. The effects of 
sleep loss on cognitive performance are primarily 
manifested as increasing variability in cognitive 
speed and accuracy. Thus, behavioral responses be- 
come unpredictable with increasing amounts of fa- 
tigue. When this increased performance variability 
occurs as a result of sleep deprivation (acute or 
chronic, partial or total), it is thought to reflect 
state instability (Doran, Van Dongen, & Dinges, 
2001). State instability refers to moment-to-moment 
shifts in the relationship between neurobiological 
systems mediating wake maintenance and those 



profiles for lapses of sustained attention performance on 
the psychomotor vigilance task (PVT; Dorrian et al, 
2005); decreases in performance on working memory 
and cognitive throughput tasks (digit symbol substitu- 
tion task, serial addition task); short-term memory 
(paired-recall memory); and subjective accuracy (time 
estimation). Each point represents the mean (SEM) for a 
test bout during the 88 hours of total sleep deprivation. 
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mediating sleep initiation (Mignot et al., 2002; 
Saper et al, 2001). The increased propensity for 
sleep as well as the tendency for performance to 
show behavioral lapses, response slowing, time-on- 
task decrements, and errors of commission (Doran 
et al., 2001) are signs that sleep-initiating mecha- 
nisms deep in the brain are activating during wake- 
fulness. Thus, the cognitive performance variability 
that is the hallmark of sleep deprivation (Dinges & 
Kribbs, 1991) appears to reflect state instability 
(Dorrian, Rogers, & Baynard, 2005; Durmer & 
Dinges, 2005). Sleep-initiating mechanisms repeat- 
edly interfere with wakefulness, making cognitive 
performance increasingly variable and dependent 
on compensatory measures, such as motivation, 
which cannot override elevated sleep pressure 
without consequences (e.g., errors of commission 
increase as subjects try to avoid errors of omission — 
lapses; Durmer & Dinges, 2005; Doran et al, 2001). 

Intrusions of sleep into goal-directed perfor- 
mance are evident in increases in a variety of neu- 
robehavioral phenomena: lapses of attention, sleep 
attacks (i.e., involuntary naps), increased fre- 
quency of voluntary naps, shortened sleep latency, 
slow eyelid closures and slow-rolling eye move- 
ments, and intrusive daydreaming while engaged 
in cognitive work (Dinges & Kribbs, 1991; Kleit- 
man, 1963). Remarkably, these phenomena can 
occur even in healthy sleep-deprived people en- 
gaged in potentially dangerous activities such as 
driving. Sleepiness-related motor vehicle crashes 
have a fatality rate and injury severity level similar 
to alcohol-related crashes. Sleep deprivation has 
been shown to produce psychomotor impairments 
equivalent to those induced by alcohol consump- 
tion at or above the legal limit (Durmer & Dinges, 
2005). 

Sleep deprivation degrades many aspects of 
neurocognitive performance (Dinges & Kribbs, 
1991; Dorrian & Dinges, 2005; Durmer & Dinges, 
2005; Harrison & Home, 2000). In addition, there 
are hundreds of published studies on the cognitive 
effects of sleep deprivation showing that all forms 
of sleep deprivation result in increased negative 
mood states, especially feelings of fatigue, loss of 
vigor, sleepiness, and confusion. Although feelings 
of irritability, anxiety, and depression are believed 
to result from inadequate sleep, most experimental 
settings do not find such changes owing to the 
comfortable and predictable environment in which 
subjects are studied. On the other hand, increased 



Table 14.1. Summary of Cognitive Performance 
Effects of Sleep Deprivation 

Involuntary microsleeps occur, which can lead to 

increasingly long involuntary naps. 

Performance on attention-demanding tasks, such as 

vigilance, is unstable with increased errors of omission 

and commission. 

Reaction time slows. 

Time pressure increases cognitive errors, and cognitive 

slowing occurs in subject-paced tasks. 

Working memory and short-term recall decline. 

Reduced learning (acquisition) of cognitive tasks. 

Performance requiring divergent thinking (e.g., 

multitasking) deteriorates. 

Response perseveration on ineffective solutions is 

more likely. 

Increased compensatory effort is required to remain 

behaviorally effective. 

Task performance deteriorates as task duration increases 

(e.g., vigilance). 

Neglect of activities considered nonessential (i.e., loss 

of situational awareness). 



negative mood states have been observed often 
when sleep deprivation occurs in complex real- 
world conditions. 

Sleep deprivation induces a wide range of ef- 
fects on cognitive performance (see table 14.1). In 
general, cognitive performance becomes progres- 
sively worse when time on task is extended — this is 
the classic fatigue effect that is exacerbated by sleep 
loss. However, performance on even very brief cog- 
nitive tasks that require speed of cognitive through- 
put, working memory, and other aspects of 
attention have been found to be sensitive to sleep 
deprivation. Cognitive work involving the pre- 
frontal cortex (e.g., divergent thinking) is adversely 
affected by sleep loss. Divergent skills involved in 
decision making that decrease with sleep loss in- 
clude assimilation of changing information, updat- 
ing strategies based on new information, lateral 
thinking, innovation, risk assessment, maintaining 
interest in outcomes, mood-appropriate behavior, 
insight, communication, and temporal memory 
skills (Harrison & Home, 2000). Implicit to diver- 
gent thinking abilities is a heavy reliance on execu- 
tive functions — especially working memory and 
control of attention (see also chapter 1 1 , this vol- 
ume). Working memory and executive attention in- 
volve the ability to hold and manipulate information 
and can involve multiple sensory-motor modalities. 
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Therefore, deficits in neurocognitive performance 
due to sleep loss can compromise the following 
functions that depend on good working memory: 
assessment of the scope of a problem due to chang- 
ing or distracting information, remembering the 
temporal order of information, maintaining focus 
on relevant cues, maintaining flexible thinking, 
avoiding inappropriate risks, gaining insight into 
performance deficits, avoiding perseveration on in- 
effective thoughts and actions, and making behav- 
ioral modifications based on new information 
(Durmer & Dinges, 2005). Although executive 
functions compromised by sleep deprivation are 
mediated by changes in prefrontal and related corti- 
cal areas, the effects of sleep loss likely originate in 
subcortical systems (hypothalamus, thalamus, and 
brain stem). This may explain why the most sensi- 
tive performance measure of sleep deprivation is a 
simple sustained attention task, such as the psy- 
chomotor vigilance task (PVT; Dorrian, Rogers, & 
Dinges, 2005). 

Rest Time and the Effects of Chronic 
Partial Sleep Deprivation 

Among the most problematic issues in work-rest 
regulations is the question of how much rest (time 
off work) should be mandated to ensure that work- 
ers avoid sleep deprivation. Instantiated in many 
federal regulatory work rules (e.g., in each major 
transportation modality) is the assumption that al- 
lowing 8 hours for rest between work periods will 
result in adequate recovery sleep to avoid depriva- 
tion. Virtually all studies of sleep during varying 
work-rest schemes reveal that actual physiological 
sleep accounts for only about 50-75% of rest time, 
which means that people allowed 8 hours to re- 
cover actually sleep 4-6 hours at most. Early ex- 
periments on chronic restriction of sleep to 4-6 
hours per night found few adverse effects on per- 
formance measures, but these studies lacked key 
experimental controls and relied on small sample 
sizes and limited measurements (Dinges, Baynard, 
& Rogers, 2005). More recent carefully controlled 
experiments on healthy adults found clear and dra- 
matic evidence that behavioral alertness and a 
range of cognitive performance functions involving 
sustained attention, working memory, and cogni- 
tive throughput deteriorated systematically across 
days when nightly sleep duration was between 
4 and 7 hours (Belenky et al., 2003; Van Dongen 



et al., 2003). In contrast, when time in bed for 
sleep was 8 hours (Van Dongen et al., 2003) or 9 
hours (Belenky et al., 2003), no cumulative cogni- 
tive performance deficits were found across days. 
Figure 14.2 shows comparable data from each of 
these studies. In one study, truck drivers were ran- 
domized to 7 nights of 3, 5, 7 or 9 hours of time in 
bed for sleep per night (Belenky et al, 2003). Sub- 
jects in the 3- and 5-hour time-in-bed groups ex- 
perienced a decrease in performance across days of 
the sleep restriction protocol, with an increase in 
the mean reaction time, number of lapses, and 
fastest reaction times on the PVT. In the subjects al- 
lowed 7 hours of time in bed per night, a signifi- 
cant decrease in mean response speed was also 
evident, although no effect on lapses was evident. 
Performance in the group allowed 9 hours of time 
in bed was stable across the 7 days. 

In an equally large experiment (Van Dongen 
et al., 2003), adults (mean age 28 years) had their 
sleep duration restricted to 4, 6, or 8 hours of time 
in bed per night for 14 consecutive nights. Cumu- 
lative daytime deficits in cognitive function were 
observed for lapses on the PVT, for a memory task, 
and for a cognitive throughput task. These deficits 
worsened over days of sleep restriction at a faster 
rate for subjects in the 4- and 6-hour sleep periods 
relative to subjects in the 8-hour control condition, 
which showed no cumulative performance deficits. 
In order to quantify the magnitude of cognitive 
deficits experienced during 14 days of restricted 
sleep, the findings from the 4-, 6-, or 8-hour sleep 
periods were compared with cognitive effects after 
1,2, and 3 nights of total sleep deprivation (Van 
Dongen et al., 2003). This comparison revealed 
that both 4- and 6-hour sleep periods resulted in 
cumulative cognitive impairments that quickly in- 
creased to levels found after 1,2, and even 3 nights 
of total sleep deprivation. 

These studies suggest that when the nightly 
recovery sleep period is routinely restricted to 7 
hours or less, the majority of motivated healthy 
adults develop cognitive performance impairments 
that systematically increase across days, until a 
longer duration (recovery) sleep period is provided. 
When nightly sleep periods in these two major ex- 
periments were 8-9 hours, no cognitive deficits 
were found. These data indicate that the basic prin- 
ciple of providing 8 hours for rest between work 
bouts in regulated industries is inadequate, unless 
people actually sleep 90% of the time allowed. 
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Figure 14.2. Results from two dose-response studies of chronic sleep restric- 
tion. Panel A is from Van Dongen et al. (2003). In this experiment, sleep was 
restricted for 14 consecutive nights in 36 healthy adults (mean age 30 years). 
Subjects were randomized to 4-hour (n = 13), 6-hour (n = 13) or 8-hour 
(n = 9) time in bed (TIB) at night. Performance was assessed every 2 hours 
(9 times each day) from 7:30 a.m. to 11:30 p.m. The graph shows cumulative 
increases in lapses of attention during the psychomotor vigilance task (PVT; 
Dorrian et al, 2005) per test bout across days within the 4-hour and 6-hour 
groups (p = .001), with sleep-dose differences between groups (p = .036). The 
horizontal dotted line shows the level of lapsing found in a separate experiment 
when subjects had been awake continuously for 64-88 hours. For example, by 
Day 7, subjects in the 6-hour TIB condition averaged 54 total lapses for the 
9 test trials that day, while those in the 4-hour TIB averaged 70 lapses per day. 
Panel B shows data from Belenky et al. (2003). In this experiment, sleep was 
restricted for 7 consecutive nights in 66 healthy adults (mean age 48 years). 
Subjects were randomized to 3-hour (n = 13), 5-hour (n = 13), 7-hour (n = 13), 
or 9-hour (n = 16) TIB at night. Performance was assessed 4 times each day 
from 9 a.m. to 9 p.m. As in (A), the graphs show cumulative increases in PVT 
lapses per test bout across days within the 3-hour and 5-hour groups 
(p = .001). The horizontal dotted line shows the level of lapsing found in a sep- 
arate experiment by Van Dongen et al. (2003) when subjects had been awake 
continuously for 64-88 hours. For example, by Day 7, subjects in the 5-hour 
TIB averaged 24 total lapses for the four test trials that day, while those in the 
3-hour TIB averaged 68 lapses that day. Reprinted with permission of Black- 
well Publishing from Belenky et al. (2003). 
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Since studies consistently show that people use 
only 50-75% of rest periods to sleep, it would be 
prudent to provide longer rest breaks (e.g., 10-14 
hours) to ensure that adequate recovery sleep is ob- 
tained. Work-rest rules are not the only area in 
which these new experimental data are relevant. Re- 
cent epidemiological studies have found an in- 
creased incidence of sleep-related crashes in drivers 
reporting 6 or fewer hours sleep per night on aver- 
age (Stutts, Wilkins, Scott Osberg, & Vaughn, 2003). 

Perception of Sleepiness Versus 
Performance during Sleep Deprivation 

It is commonly assumed that people know when 
they are tired from inadequate sleep and hence can 
actively avoid sleep deprivation and the risks it 
poses to performance and safety. That has not 
proven to be the case, however. In contrast to the 
continuing accumulation of cognitive performance 
deficits associated with nightly restriction of sleep 
to below 8 hours, ratings of sleepiness, fatigue, and 
alertness made by the subjects in the sleep restric- 
tion experiments did not parallel performance 
deficits (Belenky et al., 2003; Van Dongen et al, 
2003). Instead, subjects' perceptions of their fa- 
tigue and sleepiness showed little change after the 
first few days of sleep restriction. While percep- 
tions of sleepiness and fatigue did not show sys- 
tematic increases over days of sleep restriction, 
cognitive performance functions were steadily de- 
teriorating across days of restriction in a near-linear 
manner (Belenky et al., 2003; Van Dongen et al., 
2003). As a consequence, after a week or two of 
sleep restriction, subjects were markedly impaired 
and less alert, but they felt subjectively that they 
had adjusted to the reduced sleep durations. This 
suggests that people frequently underestimate the 
cognitive impact of sleep restriction and overesti- 
mate their performance readiness when sleep de- 
prived. Other experiments using driving simulators 
have found comparable results (Banks, Catcheside, 
Lack, Grunstein, & McEvoy, 2004) — people often 
do not accurately identify their performance risks 
when sleep deprived. 

Individual Differences in Response 
to Sleep Deprivation 

Although restriction of sleep periods to below 7 
hours duration results in cumulative cognitive 



performance deficits in a majority of healthy 
adults, not everyone is affected to the same degree. 
In fact, sleep deprivation not only increases per- 
formance variability within subjects (i.e., state in- 
stability) but also reveals marked performance 
differences between subjects. That is, as sleep dep- 
rivation becomes worse over time, intersubject dif- 
ferences also increase markedly. While the majority 
of people suffer neurobehavioral deficits when 
sleep deprived, there are individuals at opposite 
ends of this spectrum — those who experience very 
severe impairments even with modest sleep loss, 
and those who show few if any cognitive deficits 
until sleep deprivation is very severe. Recently, it 
has been shown that these responses are stable and 
reliable across subjects. Cognitive performance 
changes following sleep loss were traitlike, with in- 
traclass correlations accounting for a very high per- 
centage of the variance (Van Dongen, Maislin, & 
Dinges, 2004). However, as with chronic sleep re- 
striction, subjects were not really aware of their dif- 
ferential vulnerability to sleep loss. The biological 
basis of the differential responses to sleep loss is 
not known. Consequently, until objective markers 
for differential vulnerability to sleep deprivation 
can be found, it will not be possible to use such in- 
formation in a manner that reduces the risk posed 
by fatigue in a given individual. 



Operational Causes of Sleep Loss 

Work and related operational demands can affect 
the magnitude of sleep loss and fatigue. It is esti- 
mated that more than one third of the population 
suffers from chronic sleep loss (Walsh, Dement, & 
Dinges, 2005). This is partially due to society's re- 
quirement for around-the-clock operations. Indi- 
viduals are expected to be able to adjust to any 
schedule, independent of time of day, and continue 
to remain alert and vigilant. Often this forces peo- 
ple to extend their waking hours and reduce their 
sleep time. 

Night Shift Work 

Night shift work is particularly disruptive to sleep. 
Many of the 6 million full-time employees in the 
United States who work at night on a permanent or 
rotating basis experience daytime sleep disruption 
leading to sleep loss and nighttime sleepiness on 
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the job from circadian misalignment (Akerstedt, 
2003). More than 50% of shift workers complain 
of shortened and disrupted sleep and overall tired- 
ness, with total amounts of sleep loss ranging any- 
where from 2 to 4 hours per night (Akerstedt, 
2003). 

Irregular and prolonged work schedules, shift 
work, and night work are not unique to a single 
operational environment but exist in many work 
sectors. Such schedules create physiological dis- 
ruption of sleep and waking because of misalign- 
ment of the endogenous circadian clock and 
imposed work-rest schedules. Individuals are ex- 
posed to competing time cues from the day-night 
cycle and day-oriented society and are usually in- 
adequately adapted to their temporally displaced 
work-rest schedule. 

Fatigue and Drowsy Driving 

Individuals working irregular schedules are also 
more likely to have higher exposure to nighttime 
driving, increasing the chances of drowsy driving 
and decreasing the ability to effectively respond to 
stimuli or emergency situations (Braver, Preusser, 
Preusser, Baum, Beilock, & Ulmer, 1992; Stutts et 
al, 2003). Drowsy driving is particularly challeng- 
ing in the truck-driving environment. Fatigue is 
considered to be a causal factor in 20-40% of heavy 
truck crashes. Operational demands often force 
night driving in an attempt to avoid traffic and meet 
time-sensitive schedules. Such night work is a 
double-edged sword, requiring both working when 
the body is programmed to be asleep and sleeping 
when the body is programmed to be awake. In 
recognition of the safety issues surrounding drowsy 
driving, the U.S. Department of Transportation is 
actively involved in developing programs that 
would provide fatigue-tracking technologies to the 
human operator to help manage drowsy driving 
and fatigue as part of the development of an "intelli- 
gent vehicle" (Mallis et al., 2000). 

Transmeridian Travel 

Operator fatigue associated with jet lag is a concern 
in aviation. Although air travel over multiple time 
zones is considered a technological advance, it 
poses substantial physiological challenges to hu- 
man endurance. Crew members can be required to 
work any schedule, regardless of their geographical 



location, as well as crossing multiple time zones. 
As a result, flight crews can experience disrupted 
circadian rhythms and sleep loss. Studies have 
physiologically documented episodes of fatigue 
and the occurrence of uncontrolled sleep periods 
or microsleeps in pilots (Wright & McGown, 
2001). Flight crew members, unlike most passen- 
gers, remain at their destination for a short period 
of time and never have the opportunity to adjust 
physiologically to the new time zone or work 
schedule. 

Transiting time zones and remaining at the 
new destination for days with exposure to the new 
light-dark cycle and social cues does not guarantee 
a rapid realignment (phase shift) of the sleep-wake 
cycle and circadian system to the new time zone. A 
typical jet lag experience involves arriving at a des- 
tination (new time zone) with an accumulated 
sleep debt (i.e., elevated homeostatic sleep drive). 
This ensures that the first night of sleep in the new 
time zone will occur — even if it is abbreviated due 
to a wake-up signal from the endogenous circadian 
clock — but on the second, third, and fourth nights 
the person will most likely find it more difficult to 
obtain consolidated sleep because of the circadian 
disruption. As a result, the individual's sleep is not 
maximally restorative for a number of nights in a 
row, leading to increasing difficulty being alert dur- 
ing the daytime. These cumulative effects (see fig- 
ure 14.1) can be very incapacitating and can take 
1-3 weeks to fully dissipate through full circadian 
reentrainment to the new time zone. 

The effects of jet lag are also partly dependent 
on the direction of travel. Eastward travel tends 
to be more difficult for physiological adjustment 
than westward travel because eastward transit seeks 
to impose a phase advance on the circadian clock, 
while westward transit imposes a phase delay. 
Lengthening a day by a few hours is somewhat eas- 
ier to adjust to physiologically and behaviorally than 
advancing a day by the same amount of time, al- 
though adjustment to either eastward or westward 
phase shifts of more than a couple of hours is a slow 
process, often requiring at least a 24-hour period 
(day) for each time zone crossed (e.g., transiting 
six time zones can require 5-7 days) and resulting in 
a substantial but still incomplete adjustment for 
most people, assuming they get daily exposure 
to the light-dark cycle and social rhythms of the 
new environment. Moreover, the direction of flight 
does not ensure the direction of circadian phase 
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adjustment — some people physiologically phase de- 
lay to an eastward (phase advanced) flight, which 
can require many more days for physiological ad- 
justment to occur. The reasons for the direction of 
physiological shift are not well understood, but 
likely involve individual differences in circadian dy- 
namics and light entrainment responses. 

Prolonged Work Hours and Errors: 
Medical Resident Duty Hours 
as an Example 

Other transportation modalities — such as mar- 
itime, rail, and mass transit — as well as many non- 
transportation industries must manage fatigue 
from the demands of 24-hour operations. Any oc- 
cupation that requires individuals to maintain high 
levels of alertness over extended periods of time is 
vulnerable to the neurobehavioral and work per- 
formance consequences of sleep loss and circadian 
disruption. The resulting performance effects have 
the potential of compromising safety (Dinges, 
1995). However, the hazards associated with ex- 
tended duty schedules are not always apparent to 
the public. There is a lack of public as well as pro- 
fessional awareness regarding the importance of 
obtaining adequate amounts of sleep (Walsh, De- 
ment, & Dinges 2005). For example, providing 
acute medical care 24 hours a day, year round, re- 
sults in physicians, nurses, and allied health care 
providers being awake at night and often working 
for durations well in excess of 12 hours. Chronic 
partial sleep deprivation is an inherent conse- 
quence of such schedules, especially in physicians 
in training (Weinger & Ancoli-Isreal, 2002). Hu- 
man error also increases with such prolonged work 
schedules (Landrigan et al, 2004; Rogers, Hwang, 
Scott, Aiken, & Dinges, 2004). 

To address the risks of performance errors 
posed by sleep loss in resident physicians, the Ac- 
creditation Council for Graduate Medical Education 
(ACGME, 2003) imposed duty hour limits for resi- 
dent physicians. These duty limits were intended to 
reduce the risks of performance errors due to both 
acute and chronic sleep loss by limiting residents to 
80 hours work per week and by limiting a continu- 
ous duty period to 24-30 hours. They also man- 
dated 1 day in 7 free from duty averaged over 4 
weeks, and 10-hour rest opportunities between 
duty periods (ACGME, 2003). Recent objective 
studies of residents operating under these duty 



hour limits reveal significant numbers of medical 
errors and motor vehicle crashes (Barger et al., 
2005; Landrigan et al., 2004; see also chapter 23, 
this volume). It appears that work schedules that 
permit extended duty days to well beyond 16 hours 
result in sleep deprivation and substantial opera- 
tional errors, consistent with laboratory studies. 

Although increased fatigue and sleepiness on 
the job are more common for those involved in er- 
ratic or irregular shifts, when a portion of sleep or 
the total sleep period is occurring at a time not 
conducive to sleep, sleep loss can also be problem- 
atic for individuals on a "normal" schedule with 
sleep noctumally placed. As the number of hours 
awake increases, levels of fatigue increase, espe- 
cially with extended duty shifts (Rosa & Bonnet, 
1993). Therefore, increased fatigue and sleepiness 
associated with long shifts should be carefully con- 
sidered prior to implementing extended schedules. 
However, individuals often prefer a 12-hour shift 
to an 8-hour shift (Johnson & Sharit, 2001) be- 
cause it allows for more consecutive days off and 
more opportunities for social activities and family 
time, even though the result can be increased fa- 
tigue on the job and significant decrements in per- 
formance. 



Prediction and Detection of the 
Effects of Sleep Loss in 
Operational Environments 

The growth of continuous operations in govern- 
ment and industry has resulted in considerable ef- 
forts to either predict human performance based on 
knowledge of sleep- wake temporal dynamics or de- 
tect sleepiness and hypovigilance while on the job. 
Both approaches have come under intensive devel- 
opment and scrutiny in recent years, as govern- 
ments and industries struggle to manage fatigue. 

Biomathematical Models to Predict 
Performance Capability 

One approach to predicting the effects of sleep loss 
is through the development of computer algorithms 
and models that reflect the biologically dynamic 
changes in alertness and performance due to sleep 
and circadian neurobiology. These models and algo- 
rithms are being developed as scheduling tools to 
quantify the impact of underlying interaction of 
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sleep and circadian physiology on neurobehavioral 
functioning. A number of major efforts are under- 
way internationally that focus on the application of 
biomathematical modeling in a software package in 
order to: (1) predict the times that neurobehavioral 
functioning will be maintained; (2) establish time 
periods for maximal recovery sleep; and (3) deter- 
mine the cumulative effects of different work-rest 
schedules on overall performance (Mallis, Mejdal, 
Nguyen, & Dinges, 2004). 

These biomathematical models of alertness and 
performance are founded in part on the two-pro- 
cess model of sleep regulation (Achermann, 2004). 
The two-process model describes the temporal re- 
lationship between the sleep homeostatic process 
and the endogenous circadian pacemaker in the 
brain. Although the two-process model was origi- 
nally designed to be a model of sleep regulation, its 
application has been extended to describe and pre- 
dict temporal changes in waking alertness. When 
used in this manner, the model predicts that per- 
formance decreases progressively with prolonged 
wakefulness and simultaneously varies over time in 
a circadian pattern. 

As a result of U.S. Department of Defense, U.S. 
Department of Transportation, and NASA interest 
in the deployment of scheduling software in real- 
world environments, a workshop on fatigue and 
performance modeling was conducted that re- 
viewed seven models commonly cited in scientific 
literature or funded by government funding (Mallis, 
Mejdal, Nguyen, et al., 2004). Although these bio- 
mathematical models were based on the same 
sleep-wake physiological dynamics, there was con- 
siderable diversity among them in the number and 
type of input and output variables, and their stated 
goals and capabilities (Mallis, Mejdal, Nguyen, et al., 
2004). It is widely believed that such models can 
help identify less fatiguing work-rest schedules and 
manage fatigue-related risk. However, it is critical 
that they validly and reliably predict the effects 
of sleep loss and circadian desynchrony to help 
minimize fatigue-related accidents and incidents 
(Dinges, 2004). It is clear that biomathematical 
models that instantiate sleep-wake dynamics based 
in neurobiology are not yet ready for applicability. 
Current models failed to reliably predict the ad- 
verse effects of chronic sleep restriction on per- 
formance when evaluated in a double-blind test 
(Van Dongen, 2004). While biomathematical mod- 
els hold much promise as work-rest scheduling 



tools, they should not be transitioned to real-world 
environments without substantial evidence of their 
scientific validity. Similarly, their ecological validity 
relative to different real-world work scenarios also 
needs to be established. This includes the identifi- 
cation of inputs and outputs that are both relevant 
and accurate for the specific operational environ- 
ment. 

Technologies for Detecting 
Operator Hypovigilance 

Mathematical models of fatigue seek to predict per- 
formance capability based on sleep and circadian 
dynamics. In contrast, online real-time monitoring 
technologies for fatigue seek to detect sleepiness as 
it begins to occur during cognitive work. Many 
technologies are being developed to detect the ef- 
fects of sleep loss and night work on attention (i.e., 
hypovigilance). The goal is to have a real-time sys- 
tem that can alert or warn an operator of increasing 
drowsiness before a serious adverse event occurs. 
It is believed that the earlier hypovigilance is de- 
tected, the sooner an effective countermeasure can 
be implemented, thereby reducing the chance of 
serious human error. There is a need therefore to 
be able to continuously and unobtrusively monitor 
an individual's attention to task in an automated 
fashion. Technologies that purport to be effective 
in fatigue and drowsiness detection must be shown 
to meet or exceed a number of criteria (Dinges & 
Mallis, 1998). 

Any technology developed for drowsiness de- 
tection in real-world environments must be unob- 
trusive to the user and capable of calculating 
real-time measurements. Both its hardware and 
software must be reliable and its method of drowsi- 
ness and hypovigilance detection must be accurate. 
As far as algorithm development is concerned, it 
must reliably and validly detect fatigue in all indi- 
viduals (i.e., reflect individual differences) and it 
must require as little calibration as possible, both 
within and between subjects. Overall, the drowsi- 
ness devices must meet all scientific standards and 
be unobtrusive, economical, practical, and easy to 
use. A great deal of harm can be done if invalid or 
unreliable technologies are quickly and uncritically 
implemented. 

Initial validation should be tested in a con- 
trolled laboratory environment. Although valida- 
tion may be possible in some field-based studies, 
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there remains a challenge of error variance from ex- 
traneous sources. Additionally, field studies that do 
not allow for complete manipulation of the inde- 
pendent variable (e.g., a range of sleep loss) can 
mask the validity of a technology or create an ap- 
parent validity that is artificial and therefore could 
not be generalized to other contexts. 

Fatigue and Drowsiness Detection: Slow 
Eyelid Closures as an Example 

The first problem to confront in the online detec- 
tion of fatigue in an operational environment is 
determining what biological or behavioral (or 
biobehavioral) parameter to detect. What are the 
objective early warning signs of hypovigilance, fa- 
tigue, sleepiness, or drowsiness? This question is 
not yet resolved, but research sponsored by the 
U.S. Department of Transportation has helped 
identify a likely candidate measure of fatigue 
from sleep loss and night work (Dinges, Mallis, 
Maislin, & Powell, 1998). Researchers experimen- 
tally tested the scientific validity of six online 
driver-based alertness-drowsiness detection tech- 
nologies (e.g., EEG algorithms, eye blink detectors, 
head position sensor arrays). The criterion variable 
against which these technologies were evaluated 
was the frequency of lapses on the PVT — a test well 
validated to be sensitive to sleep loss and night 
work (Dorrian, Rogers, & Dinges, 2005). Results 
showed that only Perclos (a measure of the pro- 
portion of time subjects had slow eye closures; 
Weirwille, Wreggit, Kirn, Ellsworth, & Fairbanks, 
1994) was more accurate in the detection of the 
frequency of drowsiness-induced PVT performance 
lapses than were other approaches. In these 
double -blind experiments, Perclos was also supe- 
rior to subjects' own ratings of their sleepiness 
when it came to detecting PVT lapses of attention. 
Perclos has been evaluated in an over-the-road 
study of technologies for fatigue management in 
professional trucking operations (Dinges, Maislin, 
Brewster, Krueger, & Carroll, 2005). This study re- 
veals the critical role operator acceptance plays in 
both driver-based and vehicle-based measures of 
occupational fatigue. 

The human factors and ergonomic aspects of an 
operational context in which fatigue detection is 
undertaken can have a major impact on the validity 
and utility of the fatigue-detection system. This is il- 
lustrated by the application of Perclos to different 



transportation modalities. Results from implemen- 
tation research conducted in a high-fidelity truck 
simulator demonstrated that it was possible to in- 
terface an online automated drowsiness-detection 
Perclos system into the driving environment (Grace, 
Guzman, Staszewski, Mallis, & Dinges, 1998). Both 
auditory and visual feedback from the Perclos de- 
vice improved alertness and driving performance, 
especially when drivers were drowsy at night 
(Mallis et al., 2000), suggesting the system may 
promote user alertness and safety during the 
drowsiest portions of night driving. However, when 
the same automated Perclos system was tested in a 
Boeing 747-400 flight simulator to determine the 
effects of Perclos feedback on pilot alertness and 
performance during a night flight, the results were 
quite different. Unlike the truck-driving environ- 
ment, the automated Perclos system with feedback 
did not significantly counteract decrements in per- 
formance, physiological sleepiness, or mood 
(Mallis, Neri, Colletti, et al., 2004). This was largely 
due to the Perclos system having a limited field 
of view that could not capture the pilot's eyes at 
all times during the flight, due to operational re- 
quirements of constant visual scanning and head 
movements. This research demonstrates that im- 
plementation obstacles can emerge and must be 
overcome when transitioning scientifically valid 
drowsiness-monitoring technologies to an opera- 
tional environment. 



Conclusion 

Fatigue and sleepiness on the job are common oc- 
currences in today's society and result from circa- 
dian displacement of sleep-wake schedules, and 
acute and chronic sleep loss. Extensive neurobio- 
logical and neurobehavioral research have estab- 
lished that waking neurocognitive functions on the 
job depend upon stable alertness from adequate 
daily recovery sleep. Operational demands in 24- 
hour industries inevitably result in fatigue from 
sleep loss and circadian displacement, which con- 
tribute to increased cognitive errors and risk of 
adverse events — although the magnitude of the ef- 
fects can depend on the individual. Understanding 
and mitigating the risks posed by physiologically 
based variations in sleepiness and alertness in the 
workplace should be an essential function of neu- 
roergonomics. The emergence of biomathematical 
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models of temporally dynamic influences on per- 
formance capability from sleep-wake and circa- 
dian biology and the development of unobtrusive 
online technologies for detection of fatigue while 
working are two cogent examples of neuroergo- 
nomics in action. 



Main Points 

1 . Neurobiologically based circadian and sleep 
homeostatic systems interact to regulate 
changes in alertness, performance, and timing 
of sleep. 

2. Reduced sleep time results in neurobehavioral 
decrements that include increased reaction 
times, memory difficulties, cognitive slowing, 
and lapses of attention. 

3. Night work, time zone transit, prolonged 
work, and work environments that include 
irregular schedules contribute to fatigue and 
the risk it poses to safe operations. 

4. Efforts to manage fatigue and sleepiness in 
operational environments include prediction 
through biomathematical models of alertness 
and online fatigue-detection technologies. 

5. Understanding and mitigating the risks posed 
by physiologically based variations in 
sleepiness and alertness in the workplace 
should be an essential function of 
neuroergonomics . 
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Over the last 50 years, ergonomics (or human fac- 
tors) has been maturing and evolving as a unique 
and independent discipline that focuses on the na- 
ture of human-artifact interactions, viewed from 
the unified perspective of science, engineering, de- 
sign, technology, and the management of human- 
compatible systems, including a variety of natural 
and artificial products, processes, and living envi- 
ronments (Karwowski, 2005). According to the 
International Ergonomics Association (2002), er- 
gonomics is a systems-oriented discipline that 
extends across all aspects of human activity. The 
traditional domains of specialization within er- 
gonomics include physical ergonomics, cognitive er- 
gonomics, and organizational ergonomics. Physical 
ergonomics is concerned with human anatomical, 
anthropometric, physiological, and biomechanical 
characteristics as they relate to human physical 
activity. Cognitive ergonomics is concerned with 
mental processes, such as perception, memory, rea- 
soning, and motor response, as they affect interac- 
tions among humans and other elements of a 
system. Organizational ergonomics is concerned 
with the optimization of sociotechnical systems, 
including their organizational structures, policies, 
and processes. 



The discipline of ergonomics has witnessed 
rapid growth and its scope has continually ex- 
panded toward new knowledge about humans that 
can be useful in design (Karwowski, Siemionow, & 
Gielo-Perczak, 2003). The expansion in scope has 
roughly followed the sequence from physical 
(motor) to cognitive, to esthetic, and most re- 
cently to affective (emotional) factors. This in 
turn has made it necessary to consider human 
brain functioning and the ultimate supreme role 
of the brain in exercising control over human be- 
havior in relation to the affordances of the envi- 
ronment (Gibson, 1986). Most recently, the above 
developments have led to the onset of neuroer- 
gonomics, or the study of brain and behavior at 
work (Parasuraman, 2003). This chapter intro- 
duces physical neuroergonomics as the emerging 
field of study focusing on the knowledge of hu- 
man brain activities in relation to the control and 
design of physical tasks (Karwowski et al, 2003). 
We provide an introduction to this topic in sepa- 
rate sections on the human brain in control of 
muscular performance in the work environment 
and in motor control tasks. We discuss these is- 
sues in conditions of health, fatigue, and disease 
states. 
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The Human Brain in Control 
of Muscular Performance 

The insights offered by neuroscience (Zigmond 
et al., 1999) are essential to our understanding of 
the human operator functioning in complex sys- 
tems (Parasuraman, 2000). Knowledge of human 
motor control is also critical to further advances in 
occupational biomechanics in general, and to pre- 
vention of musculoskeletal disorders in industry 
due to repetitive manual tasks and material han- 
dling in particular (Karwowski et al, 2003). One 
of the important functions of the human brain is 
the control of motor activities, combined with per- 
ceptual, cognitive, and affective processes. Half a 
century ago, Sperry (1952) proposed that the main 
function of the central nervous system is the coor- 
dinated innervation of the musculature, and its 
fundamental structure and mechanisms can be un- 
derstood only on these terms. Sperry also argued 
that even for the highest human cognitive activi- 
ties, which do not require motor output, there exist 
certain essential motoric neural events. 

Recent brain imaging studies have provided 
strong support for Sperry's supposition, which has 
gained broader acceptance today. For example, ac- 
cording to Malmo and Malmo (2000), the extremely 
wide diversity of situations yielding electromyo- 
graphical (EMG) gradients suggests the possibility 
that these gradients may be universal accompani- 
ments of the organized goal-directed behavioral se- 
quences. Both motor and cognitive tasks, without 
any requirements for motor output, were found to 
produce EMG gradients. EMG gradients were not 
observed during simple, repetitive exercises. On 
the efferent side, Malmo and Malmo (2000) pro- 
posed a dual model for the production of EMG gra- 
dients, which is based on empirical findings that 
reflect the complex relations between EMG gradi- 
ent steepness and mental effort. The authors pro- 
vided the evidence for movement-related brain 
activity generated by proprioceptive input, in rela- 
tion to different types of feedback to the central 
nervous system during tasks that produce EMG 
gradients. 

The Human Motor System 

The human motor system consists of two interact- 
ing parts, peripheral and central (Wise & Shad- 
mehr, 2002; see also chapter 22, this volume). The 



peripheral motor system includes muscles and 
both motor and sensory nerve fibers. The central 
motor system has components throughout the cen- 
tral nervous system (CNS), including the cerebral 
cortex, basal ganglia, cerebellum, brain stem, and 
spinal cord. Wise and Shadmehr (2002) proposed 
that the various components of the motor system 
work as an integrated neural network and not as 
isolated motor centers. According to Thach (1999), 
the spinal cord serves as the central pattern genera- 
tor for reflexes and locomotion. As opposed to a 
sensory system, interruption of a motor system 
causes two abnormal functions: the inability to 
make an intended movement, and the spontaneous 
production of an unintended posture or move- 
ment. One distinct characteristic of the motor sys- 
tem is that many of its parts are capable of 
independently generating movement when cut off 
from other parts. In this way, each part of the sys- 
tem is a central pattern generator of movement. 

Human motor systems can also be classified 
into three interrelated subsystems: skeletal, auto- 
nomic, and neuroendocrine, which are hierar- 
chically organized (Swanson, Lufkin, & Colman, 
1999). For example, the lowest level of skeletal 
motor systems consists of the alpha motor neurons 
that synapse on skeletal muscle fibers. The next 
higher level consists of the motor pattern genera- 
tors (MPGs), which form the circuitry of interneu- 
rons that innervate unique sets of motor neuron 
pools. The highest level consists of motor pattern 
initiators (MPIs), which recognize specific input 
patterns and project to unique sets of MPGs. Swan- 
son et al. (1999) also argued that a hierarchical or- 
ganization exists between MPGs and MPIs. In this 
hierarchical model, at the lowest level (1), pools of 
motor neurons (MNs) innervate individual muscles 
that generate individual components of behavior. 
Pools of interneurons as MPGs at the next higher 
level (2), innervate specific sets of motor neuron 
pools. At the highest level (3), MPIs innervate spe- 
cific sets of MPGs. Complex behaviors are produced 
when MPIs receive specific patterns of sensory, in- 
trinsic, and cognitive inputs. 

A Hierarchical Model of the 
Human Motor System 

The main functions of the human nervous system 
can also be understood by analyzing the structural 
organization of the functional subsystems. This 
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Figure 1 5.1 . Basic information processing in the hierarchical motor system (after Swanson et al., 1999). 



approach also provides the circuit diagram of infor- 
mation processing in the nervous system. Swanson 
et al. (1999) proposed a model of basic information 
processing (figure 15.1) which assumes that behav- 
ior is determined by the motor output of the CNS 
and that motor output is a function of three inputs: 
sensory, cognitive, and intrinsic. According to 
Swanson et al. (1999), the relative importance of 
these inputs in controlling motor output varies 
from species to species and from individual to indi- 
vidual. This model postulates that human behavior 
(B) is determined by the motor system (M), which 
is influenced by three neural inputs: sensory (S), in- 
trinsic (I), and cognitive (C). Sensory inputs lead to 
reflex responses (r); cognitive inputs produce vol- 
untary responses (v); and intrinsic inputs act as 
control signals (c) to regulate the behavioral state. 
Motor system outputs (1) produce behaviors whose 
consequences are monitored by sensory feedback 
(2). The above model also shows that the cognitive, 
sensory, and intrinsic systems are interconnected. 

For example, motor control functioning and 
the utilization of processes traditionally considered 
as strictly cognitive can be illustrated with the 
model of neuronal control of precision grip. This 
model is based on extensive research of the neu- 
ronal implications of precision grip and describes 



the control of grip and load forces in a lifting task 
(Johanson, 1998). Adaptation of the fingertip 
forces to the physical characteristics of the lifted 
object involves two control mechanisms: (1) antic- 
ipatory parameter control, and (2) discrete-event 
sensory-driven control. Anticipatory parameter 
control determines appropriate motor programs 
that generate distributed muscle commands to the 
muscle exerting the fingertip forces. The specifica- 
tion of the parameters for motor commands is 
based on the important properties of the object that 
are stored in sensorimotor memory representation 
and were acquired during the previous experience. 
The discrete-event sensory-driven control uses so- 
matosensory mechanisms that monitor the progress 
of the task. When a movement is made, the motor 
system predicts the sensory input and it is compared 
to the actual sensory input produced by the move- 
ment (Schmitz, Jenmalm, Ehrosson, & Forssberg, 
2005). If there is a mismatch between the predicted 
and the actual sensory input, the somatosensory sys- 
tem triggers reactive preprogrammed patterns of 
corrective responses that modify the forces in the 
ongoing lift. The information from mismatch is 
used to update the sensorimotor representa- 
tions of the parameters for specific object (see 
figure 15.2). 
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The Human Brain and 
the Work Environment 

Contemporary work environments often demand 
effective human control, predictions, and decisions 
in the presence of uncertainties and unforeseen 
changes in work system parameters (Karwowski, 
1991, 2001). The description of human operators 
who actively participate in purposeful work tasks 
in a given environment and their performance on 
such tasks should reflect the complexity of brain 
activity, which includes cognition and the dynamic 
processes of knowing. Bateson (2000) suggested 
that the processes of knowing were related to per- 
ception, communication, coding, and translation. 
However, he also provided a differentiation of logi- 
cal levels, including the relationship between the 
knower and the known, knowledge looping back 
as knowledge of an expanded self. 

Many control problems at work arise from a 
lack of attention to the interactions among different 
human system components in relation to affor- 
dances of the environment (Karwowski et al., 2003). 
Affordances, as opportunities for action for a par- 
ticular organism, can offer both positive (benefits) 
and negative (injury) effects (Gibson, 1986). Thus, 



affordances are objective in the sense that they are 
fully specified by externally observable physical re- 
ality, but are subjective in the sense of being de- 
pendent on the behavior of a particular kind of 
organism. Gibson suggested that perception of the 
world is based upon perception of affordances, or 
recognizing the features of the environment that 
specify behaviorally relevant interactions. 

Human consciousness at work is manifested in 
several brain activities, including thought, percep- 
tion, emotion, will, memory, and imagination (Para- 
suraman, 2000). Therefore, tools for predicting 
human performance that take into account human 
emotions, imagination, and intuition with refer- 
ence to affordances of the environment are needed 
(Gielo-Perczak & Karwowski, 2003). For example, 
Picard (2000) pointed out that in the human brain, 
a critical part of our ability to see and perceive is 
not logical but emotional. Emotions are integrated 
in a hierarchy of neurological processes that influ- 
ence brain perceptual functions. There is behav- 
ioral and physiological evidence for the integration 
of perception and cognition with emotions (see 
also chapters 12 and 18, this volume). Emotions 
define the organisms dynamic structural pattern 
and their interactions may lead to specific responses 
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in a work system (environment). Furthermore, the 
findings reported by Gevins, Smith, Mcenvoy, and 
Yu (1997), based on high-density electroen- 
cephalograph (EEG) mapping, suggest that the an- 
terior cingulate cortex is a specialized area ol the 
neocortex devoted to the regulation of emotional 
and cognitive behavior. It was also noted that when 
environmental regularities are allowed to take part 
in human behavior, they can give it coherence 
without the need of explicit internal mechanisms 
for binding perceptual entities. 



Human Brain Activity in 
Motor Control Tasks 

Motor control studies originated in 1950s when 
motor activity-related cortical potential (MRCP) 
was first described as EEG-derived brain potential 
associated with voluntary movements. Bates (1951) 
first recorded MRCP from the human scalp during 
voluntary hand movements using a crude photo- 
graphic superimposition technique. Subsequent 
successful studies were reported by Kornhuber and 
Deecke (1965) who, based on the characteristics of 
the MRCP recording, described a slowly rising neg- 
ative potential, known as the readiness potential, that 
precedes a more sharply rising negative potential, 
known as the negative slope. The onset of both the 
readiness potential and negative slope occurs prior 
to the onset of the voluntary movement and hence, 
both are considered to indicate involvement of the 
underlying brain cortical fields in preparing for the 
desired movement (Kornhuber & Deecke, 1965). 

Recently, there have been numerous studies 
showing relevance of human motor control in re- 
lation to human performance on physical tasks. 
Explanation of the neural mechanisms underlying 
voluntary movements using MRCP requires an 
understanding of the relationship between the 
magnitude of MRCP and muscle activation. The re- 
lationship between magnitude of MRCP and rate of 
force development has not been well explored. A 
systematic study of these relationships (MRCP ver- 
sus force, MRCP versus rate of force development) 
is still needed. Due to the advancements in brain 
imaging technology in recent years and the nonin- 
vasive feature of the surface EEG and MEG record- 
ings, the number of studies of brain function 
(especially motor function) involving MRCP mea- 
surements is rapidly increasing. 



Studies of Muscle Activation 

In general, many motor actions are accomplished 
without moving the body or a body part (e.g., iso- 
metric contractions). Siemionow, Yue, Ran- 
ganathan, Liu, & Sahgal (2000) investigated the 
relationship between EEG-derived MRCP and vol- 
untary muscle activation during isometric elbow- 
flexion contractions. Thus, MRCP in this study 
represents motor activity-related cortical potential 
as opposed to movement-related cortical potential. 
In one session, subjects performed isometric 
elbow-flexion contractions at four intensity levels 
(10%, 35%, 60%, and 85% maximal voluntary 
contraction or MVC). In another session, a given 
elbow-flexion force (35% MVC) was generated at 
three different rates (slow, intermediate, and fast). 
EEG signals were recorded from the scalp overly- 
ing the supplementary motor area (SMA) and con- 
tralateral sensorimotor cortex, and EMG signals 
were recorded from the skin surface overlying the 
belly of the biceps brachii and brachioradialis mus- 
cles during all contractions. The study results 
showed that the magnitude of MRCP from both 
EEG recording locations (sensorimotor cortex and 
SMA) was highly correlated with elbow-flexion 
force, rate of rising of force, and muscle EMG sig- 
nals. Figure 15.3 illustrates the relationship be- 
tween MRCP and muscle EMG across four levels of 
force. In figure 15. 3A, MRCP from the SMA site 
was compared with the EMG recorded from the bi- 
ceps brachii muscle; in (B), the MRCP (from SMA) 
was compared with the brachioradialis EMG; in 
(C), MRCP from the motor cortex site was com- 
pared with the EMG of the biceps brachii; and in 
(D), the MRCP (from motor cortex) was compared 
with the brachioradialis EMG. Data were recorded 
at the four force levels, but the EMG data at each 
force level are expressed as actual percentage of 
MVC EMG. These results suggest that MRCP repre- 
sents cortical motor commands that scale the level 
of muscle activation. 

Eccentric and Concentric Muscle Activities 

Since different nervous system control strategies 
may exist for human concentric and eccentric 
muscle contractions, Fang, Siemionow, Sahgal, 
Xiong, and Yue (2001) used EEG-derived MRCP 
to determine whether the level of cortical activa- 
tion differs between these two types of muscle 
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Figure 15.3. Relationship between motor activity-related cortical potential (MRCP) and muscle electromyograph 
(EMG) across four levels of force (after Siemionow et al., 2000). Each symbol represents a subject (n = 8). 



activities. Subjects performed voluntary eccentric 
and voluntary concentric elbow flexor contrac- 
tions against a load equal to 10% body weight. 
Surface EEG signals from four scalp locations 
overlying sensorimotor-related cortical areas in the 
frontal and parietal lobes were measured along 
with kinetic and kinematic information from the 
muscle and joint. The MRCP was derived from the 
EEG signals of the eccentric and concentric mus- 
cle contractions. Although the load supported by 
the subject was the same between the two tasks, 
the force increased during concentric and de- 
creased during eccentric contractions from the 
baseline (isometric) force. The results showed that 



although the elbow flexor muscle activation 
(EMG) was lower during eccentric than concen- 
tric actions, the amplitude of two major MRCP 
components — one related to movement planning 
and execution and the other associated with feed- 
back signals from the peripheral systems — was 
significantly greater for eccentric than for concen- 
tric actions. The MRCP onset time for the eccen- 
tric task occurred earlier than that for the 
concentric task. The authors concluded that the 
greater cortical signal for eccentric muscle actions 
suggests that the brain probably plans and pro- 
grams eccentric movements differently from the 
concentric muscle tasks. 
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Mechanism of Muscular Fatigue 

A limited number of studies regarding cortical 
modulation of muscle fatigue have been reported. 
Liu et al. (2001) investigated brain activity during 
muscle fatigue using the EEG system. Subjects per- 
formed intermittent handgrip contractions at 30% 
(300 trials) and 100% (150 trials) MVC levels. 
Each 30% contraction lasted 5 seconds and each 
100% contraction 2 seconds, with a 5-second rest 
period between adjacent contractions. EEG data 
were recorded from the scalp during all contrac- 
tions along with handgrip force and muscle EMG 
signals. MRCP was derived by force-triggered aver- 
aging of EEG data from each channel over each 
block (30 trials) of contractions with MRCP ampli- 
tude quantified. Thus, for each channel there were 
5 MRCP data points for the 100% level task and 10 
such data points for the 30% level task after aver- 
aging. Each data point represented cortical activity 
corresponding to a unique fatigue status of the 
subject or time frame (e.g., the first and last points 
corresponded respectively to conditions of least 
and most fatigue). Fatigue was determined by eval- 
uating changes in force (100% level task) and EMG 
(30% level task), which declined to about 40%, 
while EMG of the flexor muscles (i.e., FDP and 
FDS) declined to about 45% of the maximal levels 
in 400 seconds. The results showed that the hand- 
grip force and EMG decreased in parallel for the 
100% level task. For the 30% level task, the EMG 
of the finger flexors increased progressively while 
the force was maintained. The MRCP data, how- 
ever, did not couple closely with the force and 
EMG signals, especially for the 100% level task. It 
was concluded that the uncoupling of brain and 
muscle signals may indicate cortical and sensory 
feedback modulation of muscle fatigue. 

Chronic fatigue syndrome (CFS) is a contro- 
versial condition in which people report malaise, 
fatigue, and reduction in daily activities, despite 
few or no physiological or laboratory findings. To 
examine the possibility that CFS is a biological ill- 
ness involving pathology of the CNS, Siemionow 
et al. (2001) investigated whether brain activity of 
CFS patients during voluntary motor activities dif- 
fers from that of healthy individuals. Eight CFS pa- 
tients and eight age- and sex-matched healthy 
volunteers performed isometric handgrip contrac- 
tions at 50% MVC level. In the first experiment, 
they performed 60 contractions with a 10-second 



rest between adjacent trials — the nonfatigue (NF) 
task. In the second experiment, the same number 
of contractions was performed with only a 5- 
second rest period — the fatigue (FT) task; 64 chan- 
nels of surface EEG were recorded simultaneously 
from the scalp. Depicted data were recorded dur- 
ing fatiguing tasks from a CFS patient and a control 
subject (Ctrl). The amplitude of MRCP for the NF 
task was greater for the patient group than the con- 
trol group. Similarly, MRCP for the FT task was 
greater for the patients than for the healthy sub- 
jects. Spectrum analysis of the EEG signals indi- 
cated that there were substantial differences at the 
delta and theta frequency bands between the two 
groups. The study results support the notion that 
CFS involves alterations of the CNS system. 

Motor Control in Human Movements 

Control of Extension and Flexion Movements 

Corticospinal projections to the motor neuron pool 
of upper-limb extensor muscles have been reported 
to differ from those of the flexor muscles in humans 
and other primates. The influence of this difference 
on the CNS control for extension and flexion move- 
ments was studied by Yue, Siemionow, Liu, Ran- 
ganathan, and Sahgal (2000). Cortical activation 
during thumb extension and flexion movements of 
eight human volunteers was measured using func- 
tional magnetic resonance imaging (fMRI), which 
detects signal changes caused by an alteration in the 
local blood oxygenation level. The amplitude of 
MRCP was recorded during the thumb flexion and 
extension. The amplitude of MRCP recorded during 
thumb extension was significantly higher than that 
during flexion at both the motor cortex and SMA 
(p < .05, paired t test). Although the relative activity 
of the extensor and flexor muscles of the thumb 
was similar, the brain volume activated during ex- 
tension was substantially larger than that during 
flexion. These fMRI results were confirmed by mea- 
surements of EEG-derived MRCP It was concluded 
that the higher brain activity during thumb exten- 
sion movement may be a result of differential corti- 
cospinal and possibly other pathway projections to 
the motor neuron pools of extensor and flexor mus- 
cles of the upper extremities. 

Power and Precision Crip 

Over the last decade, a large number of neurophysi- 
ological studies have focused on investigating the 
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neural mechanisms controlling the precision grip 
(Fagergren, Ekeberg, & Forssberg, 2000; Kinoshita, 
Oku, Hashikawa, & Nishimura, 2002; Schmitz 
et al., 2005). These studies analyzed how a human 
manipulates an object with the tips of the index fin- 
ger and thumb, or compensates for sudden pertur- 
bation when holding the object. The studies, using 
cell recording and different brain imaging methods, 
established a close relationship between primary 
motor cortex and precision grip (Ehrsson et al., 
2000; Lemon, Johanson, & Westling, 1995; Salimi, 
Brochier, & Smith, 1999). The activation of brain 
areas during performance of precision grip in hu- 
mans was also investigated in reference to several 
factors, including the following: different force size 
applied to object during the grip (Ehrsson, Fager- 
gren, & Forssberg, 2001), object weight changes 
(Kinoshita et al, 2000; Schmitz et al., 2005), differ- 
ent types of object surface texture and friction char- 
acteristics (Salimi, 1999), and different types of grip 
(precision vs. power grip; Ehrsson et al., 2000). 

Kilner, Baker, Salenius, Hari, and Lemon (2001) 
investigated task-dependent modulation incoherence 
between motor cortex and hand muscles during pre- 
cision grip tasks. Twelve right-handed subjects used 
index finger and thumb to grip two levers that were 
under robotic control. Each lever was fitted with a 
sensitive force gauge. Subjects received visual feed- 
back of lever force levels and were instructed to keep 
them within target boxes throughout each trial. Sur- 
face EMGs were recorded from four hand and fore- 
arm muscles, and magnetoencephalography (MEG) 
was recorded using a 306-channel neuromagne- 
tometer. Overall, all subjects showed significant lev- 
els of coherence (0.086-0.599) between MEG and 
muscle in the 15-30 Hz range. Coherence was sig- 
nificantly smaller when the task was performed 
under an isometric condition (levers fixed) com- 
pared with a compliant condition in which sub- 
jects moved the levers against a springlike load. 
Furthermore, there was a positive, significant rela- 
tionship between the level of coherence and the 
degree of lever compliance. These results argue in 
favor of coherence between cortex and muscle be- 
ing related to specific parameters of hand motor 
function. 

Ehrsson et al. (2000) used fMRI to analyze hu- 
man brain activity during performance of two dif- 
ferent kinds of manual grip: power grip and 
precision grip. The power grip is a palmar opposi- 
tion grasp in which all digits are fixed around the 



object. Results showed that application of the pre- 
cision grip was associated with a different pattern 
of brain activity in comparison to the power grip. 
When the power grip was performed, the activity 
of the primary sensory and motor cortex contralat- 
eral to the operating hand was higher than when 
subjects performed the precision grip. The preci- 
sion grip task was more strongly associated with 
activity in the ipsilateral ventral premotor area, the 
rostral cingulate motor area, and at several loca- 
tions in the posterior parietal and prefrontal cor- 
tices. Besides, it was also found that while the 
precision grip involved extensive activation in both 
hemispheres, the power grip was mainly related to 
activity in the left hemisphere. 

Kinoshita (2000) investigated regional cortical 
and subcortical activation induced by repetitive 
lifting using a precision grip between index finger 
and thumb with positron emission tomography 
(PET). Results revealed significant activation of the 
primary motor (Ml), primary sensory (SI), dorso- 
caudal premotor (PM), caudal SMA, and cingulate 
motor (CMA) cortices contralateral to the hand 
used during the object lifting. Behavioral adapta- 
tions to a heavier object weight were reflected in a 
nearly proportional increase of grip and lift forces, 
prolonged force application period, and a higher 
level of hand and arm muscle activity. An increase 
of regional cerebral blood flow (rCBF) that can be 
associated with these changes was found in several 
cortical and subcortical areas. However, consistent 
object weight-dependent activation was noted only 
in the Ml/Si contralateral to the hand used. 

Ehrsson et al. (2001) conducted an fMRI study 
on human subjects which investigated if the corti- 
cal control of small precision grip forces differs 
from control of the large grip forces when the same 
grasp is applied to a stationary object. The research 
revealed that several sensory- and motor-related 
frontoparietal areas were more strongly activated 
when subjects applied a small force in comparison 
to when they generated a larger force. The fMRI 
showed brain regions with significant increased ac- 
tivity when the subjects applied small fingertip 
forces as compared with large forces. It was con- 
cluded that secondary sensorimotor areas in the 
frontal and parietal lobes of the human brain play 
an important role in the control of fine precision- 
grip forces in the small range. The stronger activa- 
tion of those areas during small force application 
may reflect the involvement of additional sensori- 



Physical Neuroergonomics 



229 



motor mechanism control and somatosensory in- 
formation processing due to more demanding con- 
trol of fine object manipulation. 

Reaching Movements 

Neuroimaging studies (Grossman et al., 2000) point 
to the existence of neural mechanisms specialized 
for analysis of the kinematics defining motion. In or- 
der to reach an object, the spatial information about 
a target from the sensory system should be trans- 
formed into the motor plan. Studies have shown 
that the transformation of this information is per- 
formed in the posterior parietal cortex (PPC), since 
it is placed between visual areas that encode spatial 
information and motor cortical areas (Cloweer et al., 
1996; Kalaska, Scott, Cisek, & Sergio, 1997; Kerzt- 
man, Schwarz, Zeffiro, & Hallt, 1997; Sabes, 2000; 
Snyder, Batista, & Andersen, 1997). 

Snyder et al. (1997) analyzed neuronal activity 
in PPC during the early planning phases of the 
reaching task in the monkey and during saccades 
to the single remembered location. The pattern of 
neuronal activities before the movement depended 
on the type of movement being planned. The activ- 
ity of 373 out of 652 recorded neurons was signifi- 
cantly modulated by direction of movement during 
the tasks. Also, 68% of these neurons were motor- 
intention specific: 21% were significantly modu- 
lated before eye but not arm movement, and 47% 
were significantly modulated before arm but not 
eye movement. Thus it has been suggested that 
there are two separate pathways in the PPC area, 
one related to reach intention and another related 
to saccade intentions. 

Kerztman et al. (1997) conducted a PET study 
on visual guided reaching with the left or right arm 
to the targets presented in either the right or left vi- 
sual fields. Two separate effects on neural activity 
were analyzed: (1) hand effect, the hand used to 
reach irrespective of field reach; and (2) the. field ef- 
fect, the effect of the target field of reach irrespec- 
tive of the hand used. Significant rCBF increases in 
the hand and field conditions occurred bilaterally 
in the supplementary motor area, premotor cortex, 
cuneus, lingual gyrus, superior temporal cortex, 
insular cortex, thalamus, and putamen. Based on 
the results, Kerztman et al. (1997) suggested that 
the visual and motor components of reaching have 
different functional organizations and many brain 
areas represent both reaching limb and target field. 
The authors also observed a close correspondence 



in identified activation areas for both effects. The 
comparison of results for both conditions did not 
show separated regions related either to hand or vi- 
sual field. Besides, the posterior parietal cortex is 
related to all areas identified in the study; thus it 
has been concluded that it plays a main role in the 
integration of the limb and visual field information. 
The control of grasping and manipulations re- 
lies on the distributed processes in the CNS that 
involve most areas related to the sensorimotor con- 
trol (Lemon et al., 1995; Schmitz et al., 2005). The 
motor cortex, via descending influences over the 
spinal cord, modulates activity in all motoneuron 
pools involved in reach and grasp. It is believed 
that the cortico-motoneuron is particularly impor- 
tant for the execution of skilled hand tasks (Lemon 
et al., 1995). The influence of the corticospinal sys- 
tem on motor output during the reach, grasp, and 
lift of the object in human subjects was investi- 
gated with application of transcranial magnetic 
stimulation (TMS; Lemon et al., 1995). It was hy- 
pothesized that the influence of the motor cortex 
on the hand muscles should appear during the 
movement phases that place high demands on the 
sensorimotor control, that is, during the position- 
ing of the fingers prior to touch and during the 
early lifting phases. TMS was directed at the hand 
area of the motor cortex and was delivered during 
the six phases: reaching, at grip close, object ma- 
nipulation, parallel increase in grip and load forces, 
lifting movement, and holding of the object. It was 
concluded that observed modulation in the EMG 
responses across different phases of the task 
evoked by the TMS reflect phasic changes in corti- 
cospinal excitability. This conclusion was based on 
the observation that for each muscle clear dissocia- 
tions were found between the changes in the am- 
plitude of the response to the TMS and the 
variations of EMG related to the different phases of 
the task. The obtained results suggest that the cor- 
tical representations of the intrinsic muscles that 
control the hand and fingertip orientation were 
highly activated when the digits closed around the 
object and just after the subject touched the object. 
The extrinsic muscles that control orientation of 
the hand and fingertips received the largest cortical 
input during the reaching phase of the task. 

Postural Adjustments and Control 

One of the unresolved problems in motor control 
research is whether maintaining fixed limb posture 
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and movement between postures involves single 
robust or different specialized neural control pro- 
cesses (Kurtzer, 2005). Both approaches have their 
own advantages and disadvantages. The single con- 
trol process may allow generality and conserved 
neural circuitry, which can be illustrated with equi- 
librium point control. The specialized control pro- 
cesses allow for context dependency and flexibility. 
Kurtzer analyzed this problem by investigating mo- 
tor cortex neuron activity representation of me- 
chanical loads during posture and reaching tasks. 
It was observed that the activity of approximately 
half of the neurons that reflected load-related activ- 
ity was related specifically to either posture or 
movement only. Those neurons that were activated 
during both tasks randomly switched their response 
magnitude between tasks. Kurtzer suggested that 
the observed random changes in load representa- 
tion of neuronal activity provide evidence for dif- 
ferent specialized control processes of posture and 
movement. The existence of a switch in neural 
processing of the motor cortex just before the 
transition from stationary posture to body move- 
ment provides support for task-dependent control 
strategies. 

Kazennikov, Solopova, Talis, Grishin, and Ioffe 
(2005) investigated the role of the motor cortex in 
the anticipatory postural adjustments for forearm 
unloading. In this study, motor evoked potentials 
(MEPs) evoked by TMS were analyzed in a forearm 
flexor at the time of bimanual unloading. MEPs 
were recorded in the forearm flexor during differ- 
ent tasks, including active and passive unloading, 
static forearm loading, and static loading of one 
hand with simultaneous lifting of the same weight 
by the other hand. Anticipatory postural adjust- 
ments consisted of changes in the activity of a fore- 
arm flexor muscle prior to active unloading of the 
limb and acted to stabilize the forearm position. It 
was found that during active and passive unload- 
ing, MEP amplitude decreased with the decrease of 
muscle activity. During stationary forearm loading, 
the change in MEP corresponded to the degree of 
loading. If during static loading the contralateral 
arm has lifted a separate, equivalent weight, the 
amplitude of MEP decreased. No specific changes 
in cortical excitability were found during the antic- 
ipatory postural adjustment. Kazennikov et al. sug- 
gested a possibility of direct corticospinal volley 
and the motor command mediated by subcortical 
structures in anticipatory postural adjustments. 



Also, based on the results of several previous stud- 
ies, it was suggested that the motor cortex plays a 
predominant role in learning a new pattern of pos- 
tural adjustments, but not in physical perfor- 
mance. 

Motor Control Task Difficulty 

Winstein, Grafton, and Pohl (1997) analyzed brain 
activity related to motor task difficulty during per- 
formance of goal-directed arm aiming using PET. 
The study used and applied Fitts continuous aim- 
ing paradigm with three levels of difficulty and two 
aiming types (transport vs. targeting). Kinematics 
characteristics and movement time were analyzed 
along with the magnitude of brain activity in order 
to determine the brain areas related to the task and 
movement variables. The results showed signifi- 
cant differences in rCBF in reference to different 
task conditions. Reciprocal aiming compared with 
no-movement conditions resulted in significant 
differences in brain activity in the following areas: 
the cortical areas in the left sensorimotor, dorsal 
premotor, and ventral premotor cortices, caudal 
SMA proper, and parietal cortex, and subcortical 
areas of the left putamen, globus pallidus, red nu- 
cleus, thalamus, and anterior cerebellum. These 
brain areas are associated with the planning and 
execution of goal-directed movements. The in- 
crease of task difficulty gave the increase of rCBF in 
areas associated with planning complex move- 
ments requiring greater visuomotor processing. 
The decrease of task difficulty resulted in signifi- 
cant increases of brain activity in the areas related 
to high motor execution demands and minimal de- 
mands for precise trajectory planning. 

This study presented an interesting testing of 
Fitts' law. According to Fitts' law, movement time 
increases as the width of the target decreases or 
when the distance to the target increases (Fitts, 
1954). This time dependency describes the speed- 
accuracy trade-off for rapid, goal-directed aiming 
movements. The increase in movement time was 
explained as result of increased motor planning de- 
mands and the use of visual feedback to maintain 
accuracy with increase task difficulty. Winstein et 
al. (1997) stated that obtained brain activation pat- 
terns in this study support both arguments. The 
observed activation increase in bilateral fusiform 
gyrus suggests increase in visual information pro- 
cessing. The results also showed that with an in- 
crease in task difficulty, the activation of three areas 
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of frontal cortex increases. These areas are associ- 
ated with motor planning. When the aiming task 
required shorter-amplitude movements with pre- 
cise endpoint constraints, the results showed in- 
creased activity in the dorsal parietal area and left 
central sulcus. According to Winstein et al. (1997), 
these results suggest that contrary to Fitts' law pre- 
dictions, the brain areas' activity pattern reflects in- 
creased targeting demands. 

Load Expectation 

Schmitz et al. (2005) investigated brain activity re- 
lated to the sensorimotor memory representation 
during predictable and unpredictable weight 
changes in the lifting task. During repetitive object 
lifting, the fingertip forces are targeted to the 
weight of the objects. The anticipatory program- 
ming depends on the sensorimotor memory repre- 
sentation that possesses information on the object 
weight. The unpredicted changes of object weight 
lead to mismatch between the predicted and actual 
sensory output related to the object weight. This in 
turn triggers corrective mechanisms and updating 
of the sensory memory. In this study, 12 subjects 
lifted an object with the right index finger and 
thumb in three conditions: constant weight, regu- 
lar weight change, and irregular weight change. Re- 
sults obtained with fMRI showed that some of the 
cerebral and subcortical brain areas related to the 
precision grip lift were more active during irregular 
and regular lifting than during constant weight 
conditions. The larger activation of the three corti- 
cal regions (left parietal operculum, bilateral supra- 
marginal gyrus, and bilateral inferior frontal 
cortex) may be related to the fact that weight 
changing conditions caused errors in motor pro- 
gramming that triggered corrective reactions and 
updating of sensory memory representation. Some 
of these areas were more activated during the irreg- 
ular than regular change of weight. It was sug- 
gested that larger activation during the irregular 
weight change may be due to larger errors occur- 
ring in the programming of the fingertip forces 
during unexpected weight changes. 

Internal Models and Motor Learning 

Research on motor control shows the existence of 
representations of sensorimotor transformations 
within the CNS and close covariation of motor cor- 



tex with arm postures, output force, and muscle ac- 
tivity This evidence was used to support the theoret- 
ical framework, claiming that a nervous system uses 
internal models in controlling complex movement 
(Kalaska et al., 1997; Kawato, Furukawa, & Suzuki, 
1987; Sabes, 2000). Internal models of the motor 
system are hypothetical computations within the 
nervous system that simulate the naturally occurring 
transformations between the sensory signals and 
motor commands (Witney, 2004). Two main types of 
internal models are distinguished: forward models 
that predict the outcome of some motor event and 
inverse models or feedforward controllers that calcu- 
late the motor command required to achieve some 
desired state. In the case of visually guided move- 
ment, it is believed that the early stages of a reaching 
movement are under control of the forward internal 
model, whereas the later stages are under control of 
a feedforward model (Sabes, 2000). 

Neuroimaging studies have revealed that learn- 
ing a new motor or cognitive task causes an in- 
crease of regional blood flow in the human 
cerebellum at the beginning of the learning pro- 
cess, which decreases as the learning process pro- 
gresses. Nezafat, Shadmehr, and Holcomb (2001) 
used PET to examine changes in the cerebellum as 
subjects learned to make movements with their 
right arm while holding the handle of a robot that 
produced a force field. They observed that motor 
errors decreased from the control condition to the 
learning condition. With practice, initially dis- 
torted movements became smooth, straight-line 
trajectories. Shadmehr and Mussa-Ivaldi (1994) 
hypothesized that this improvement is due to for- 
mation of an internal model in the brain. Finally, 
Shadmehr and Brashers-Krug (1997) observed that 
a single session of training was sufficient to allow 
the subject to form an accurate internal model and 
maintain this model for up to 5 months. 



Discussion 

One of the most important functions of the human 
brain is the control of motor activities, combined 
with perceptual, cognitive, and affective processes. 
The studies discussed above demonstrate the com- 
plexity of the relationships between muscular per- 
formance and the nature of human brain control 
over physical activities. In addition to enhancing 
our understanding of human motor performance, 
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the results of these studies provide many important 
insights into workplace design. For example, it has 
been shown that during the isometric elbow- 
flexion contractions, the cerebral-cortex system 
controls the extent of muscle activation and is re- 
sponsible for smoothing out high-speed motor 
control processes (Siemionow et al, 2001). Sur- 
prisingly, brain activation during thumb extension 
was substantially larger than that observed during 
thumb flexion (Yue, Siemonow, Ranganatham, & 
Sahgal, 2000). It was also suggested that the higher 
brain activity might be a result of differential corti- 
cospinal projections to the motor neuron pool of 
upper-limb extensor muscles. Research on the pre- 
cision grip demonstrated that the relationships be- 
tween muscular force and human brain activity 
patterns are more complex than the proportional 
increase of brain activity with increase of force ap- 
plied (Kinoshita et al, 2000). The activation in 
some brain areas significantly increased when 
small forces were applied at the fingertip-object in- 
terface (Ehrsson et al., 2001). The investigation of 
differences between brain activation patterns dur- 
ing different grip types (small force vs. large force, 
precision grip vs. power grip) demonstrated that 
precise and fine finger-object manipulations re- 
quire additional sensorimotor control mechanisms 
to control force, and are more demanding in terms 
of neural control (Ehrsson et al., 2000, 2001). The 
results of eccentric versus concentric muscle con- 
tractions revealed that brain areas can be activated 
by motor imagery of those actions (Fang et al., 
2001). The influence of mental training on 
strength of muscles was also analyzed (Ran- 
ganathan, Siemionow, Sahgal, & Yue, 2001). The 
range of synchronization of beta range EEG-EMG 
revealed a state of the corticomuscular network 
when attention was directed toward the motor task 
(Kristeva-Feige, Fritsch, Timmer, & Lucking, 
2002), stipulating higher synchronization effects 
with higher precision of the motor task. The mag- 
netoencephalography study revealed the coherence 
between motor cortex and muscle activity, with 
smaller coherence for isometric tasks compared 
with the dynamic hand activities (Kilner et al., 
2001). A brain study of the affected patients versus 
healthy group confirmed that CFS involves impair- 
ments of the CNS (Siemionow et al., 2001). 

The neurophysiological research on the motor 
control of different types of complex movement 



may provide important and very specific informa- 
tion for ergonomic design concerning what impor- 
tant parameters of the object to be grasped or lifted 
are used by central neuronal processes to plan 
body movement. For example, knowledge about 
how sensory spatial information is used by the 
brain to plan the goal-directed voluntary move- 
ments may be helpful in planning and design of 
the manual work task and work place. Further- 
more, knowledge about how the neural control 
mechanism adapts the hand movement and force 
parameters to the physical properties of manipu- 
lated objects may help in developing optimal and 
specific strategies for manual handling tasks or 
other skilled hand tasks. The results of research on 
brain activity during reaching movement revealed 
the role of visual feedback at the early stages of 
reach planning and the ability to precisely control 
the complex dynamics of multijoint movements. It 
has also provided information on the role of learn- 
ing in the maintenance of internal models and the 
way in which intrinsic (e.g., joint, muscle) and ex- 
trinsic (e.g., perceptual, task specific) information 
is used and combined to form a motor action plan. 
The investigation of how the CNS utilizes spatial 
information about target locations and limb posi- 
tion for specification of movement parameters, 
such as arm position or work posture, can help in 
providing specific visual cues to guide optimal and 
safe physical performance. Finally, mathematical 
models based on the concepts of cerebellum struc- 
tures and neural control mechanisms can provide a 
wide range of possibilities to simulate and model 
human musculoskeletal responses during perfor- 
mance under a variety of working conditions, in 
order to validate the design of tools, workplaces, 
and physically demanding tasks. 



Conclusion 

As discussed by Karwowski et al. (2003), the selec- 
tive approaches taken by researchers who work in 
different domains of ergonomics and the critical 
lack of knowledge synthesis may significantly slow 
down growth of the human factors discipline. For 
example, physical ergonomics, which focuses on 
musculoskeletal system performance (biomechan- 
ics), provides very little, and inadequate at best, 
consideration of cognitive design issues. On the 
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other hand, cognitive ergonomics mostly disre- 
gards the muscular aspects of performance require- 
ments of the variety of human-machine interaction 
tasks (Karwowski, 1991; Karwowski, Lee, Jamaldin, 
Gaddie, & Jang, 1999). The physical neuroer- 
gonomics approach offers a new methodological 
perspective for the study of human-compatible sys- 
tems in relation to working environments and 
work task design. In this approach, the human 
brain is exerting control over its environment by 
constructing behavioral control networks, which 
functionally extend outside of the body, making 
use of consistent properties of the environment. 
These networks and the control they allow are the 
very reason for having a brain. In this context, hu- 
man performance can be modeled as a dynamic, 
nonlinear process taking place over the interac- 
tions between the human brain and the environ- 
ment (Gielo-Perczak & Karwowski, 2003). 

The focus on the human brain in control of 
physical activities can be a very useful tool for min- 
imizing incompatibilities between the capacities of 
workers and demands of their jobs in the context 
of affordances of the working environment. Such 
an approach can also help in assessing the suitabil- 
ity of human-machine system design solutions and 
determining the most effective improvements at the 
workplace. The exemplary results of neuroscience 
studies discussed in this chapter point out the crit- 
ical importance of our understanding of brain 
functioning in control of human tasks. Such 
knowledge is also very much needed to understand 
the mechanisms for causation of musculoskeletal 
injuries, including work-related low back or upper 
extremity disorders, and to stimulate the develop- 
ment of new theories and applications to prevent 
such disorders. 

Consideration of human motor activities sepa- 
rately from the related sensation and perception of 
our environment, information processing, decision 
making, learning, emotions, and intuition leads to 
system failures, industrial disasters, workplace in- 
juries, and many failed ergonomics interventions. 
Neuroergonomics design rejects the traditional 
perspective of divisions between physical, percep- 
tual, or cognitive activities, as they all are con- 
trolled by the brain. From this perspective, the 
functioning of the brain must be reflected in devel- 
opment of design principles and operational re- 
quirements for human-compatible systems. 



Main Points 

1. One of the most important functions of the 
human brain is the control of motor activities, 
combined with perceptual, cognitive, and 
affective processes. 

2. The studies discussed in this chapter 
demonstrate the complexity of the 
relationships between muscular performance 
and the nature of human brain control over 
physical activities. 

3. Specifically, knowledge about how the neural 
control mechanism adapts the hand 
movement and force parameters to the 
physical properties of manipulated objects 
may help in developing optimal and specific 
strategies for manual handling tasks or other 
skilled hand tasks. 

4. Mathematical models based on the concepts of 
cerebellum structures and neural control 
mechanisms allow for modeling human 
musculoskeletal responses under a variety of 
working conditions in order to validate the 
design of tools, workplaces, and physically 
demanding tasks. 

5. Knowledge of human motor control is critical to 
further advances in occupational biomechanics 
in general and prevention of musculoskeletal 
disorders in industry due to repetitive tasks and 
manual handling of loads in particular. 

6. The physical neuroergonomics approach offers 
a new methodological perspective for the 
study of human-compatible systems in 
relation to working environments and work 
task design. 
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Adaptive Automation 



We humans have always been adept at dovetailing our minds and skills to the shape of our 
current tools and aids. But when those tools and aids start dovetailing back — when our 
technologies actively, automatically, and continually tailor themselves to us just as we do to 
them — then the line between tool and human becomes flimsy indeed. 

Andy Clark, Natural-Born Cyborgs: Minds, 
Technologies and the Future of Human Intelligence (p. 7) 



Neuroergonomics has been described as the study 
of brain and behavior at work (Parasuraman, 
2003). This emerging area focuses on current re- 
search and developments in the neuroscience of in- 
formation processing and how that knowledge can 
be used to improve performance in real-world en- 
vironments. Parasuraman (2003) argued that an 
understanding of how the brain processes percep- 
tual and cognitive information can lead to better 
designs for equipment, systems, and tasks by en- 
abling a tighter match between task demands and 
the underlying brain processes. Ultimately, re- 
search in neuroergonomics can lead to safer and 
more efficient working conditions. 

Ironically, interest in neuroergonomics evolved 
from research surrounding how operators interact 
with a form of technology designed to make work 
and our lives easier — automation. In general, au- 
tomation can be thought of as a machine agent ca- 
pable of carrying out functions normally performed 
by a human (Parasuraman & Riley, 1997). For ex- 
ample, the automatic transmission in an automo- 
bile allocates the tasks of depressing the clutch, 
shifting gears, and releasing the clutch to the vehi- 
cle. Automated machines and systems are intended 
and designed to reduce task demands and work- 
load. Further, they allow individuals to increase 



their span of operation or control, perform func- 
tions that are beyond their normal abilities, main- 
tain performance for longer periods of time, and 
perform fewer mundane activities. Automation can 
also help reduce human error and increase safety. 
The irony behind automation arises from a grow- 
ing body of research demonstrating that automated 
systems often actually increase workload and cre- 
ate unsafe working conditions. 

In his book, Taming HAL: Designing Interfaces 
Beyond 2001 , Degani (2004) relates the story of an 
airline captain and crew performing the last test 
flight with a new aircraft. This was to be the second 
such test that day and the captain, feeling rather 
tired, requested that the copilot fly the aircraft. The 
test plan required a rapid takeoff, followed by en- 
gaging the autopilot, simulating an engine failure 
by reducing power to the left engine, and then 
turning off the left hydraulic system. The test flight 
started out just fine. Four seconds into the flight, 
however, the aircraft was pitched about 4 degrees 
higher than normal, but the captain continued 
with the test plan and attempted to engage the 
autopilot. Unfortunately, the autopilot did not en- 
gage. After a few more presses of the autopilot but- 
ton, the control panel display indicated that the 
system had engaged (although in reality, the 
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autopilot had not assumed control). The aircraft 
was still pitched too high and was beginning to 
lose speed. The captain apparently did not notice 
these conditions and continued with the next 
steps, requiring power reduction to the left engine 
and shutting down the hydraulic system. 

The aircraft was now flying on one engine with 
increasing attitude and decreasing speed. More- 
over, the attitude was so steep that the system in- 
tentionally withdrew autopilot mode information 
from its display. Suddenly, the autopilot engaged 
and assumed the altitude capture mode to take the 
aircraft to the preprogrammed setting of 2,000 feet, 
but this information was not presented on the 
autopilot display. The autopilot initially began low- 
ering the nose, but then reversed course. The atti- 
tude began to pitch up again and airspeed continued 
to fall. When the captain finally turned his atten- 
tion from the hydraulic system back to the instru- 
ment panel, the aircraft was less than 1,500 feet 
above ground, pitched up 30 degrees, with air- 
speed dropping to about 100 knots. The captain 
then had to compete with the envelope protection 
system for control of the aircraft. He attempted to 
bring the nose down and then realized he had to 
reduce power to the right engine in order to undo a 
worsening corkscrew effect produced by the simu- 
lated left engine failure initiated earlier. Although 
he was able to bring the attitude back down to 
zero, the loss of airspeed coupled with the simu- 
lated left engine failure had the aircraft in a 90- 
degree roll. The airspeed soon picked up and the 
captain managed to raise the left wing, but by this 
time the aircraft was only 600 feet above ground. 
Four seconds later the aircraft crashed into the 
ground, killing all on board. 

Degani (2004) discussed several factors that 
contributed to this crash. First, no one knows why 
the autopilot's altitude was preprogrammed for 
2,000 feet, but it is possible that the pilot never en- 
tered the correct value of 10,000 feet. Second, al- 
though the pilot tried several times to engage the 
autopilot, he did not realize that the system's logic 
would override his requests because his copilot's 
attempts to bring the nose down were canceling his 
requests. Third, there was a previously undetected 
flaw in the autopilot's logic. The autopilot calcu- 
lated the rate of climb needed to reach 2,000 feet 
when both engines were powered up, but did not 
recalculate the rate after the left engine had been 
powered down. Thus, the autopilot continued to 



demand the power it needed to reach the prepro- 
grammed altitude despite that the aircraft was los- 
ing speed. Last, no one knows why the pilot did 
not disengage the autopilot when the aircraft con- 
tinued to increase its attitude. Degani suggested 
that pilots who have substantial experience with 
autopilot systems may place too much trust in 
them. Thus, it is possible that assumptions regard- 
ing the reliability of the autopilot coupled with the 
absence of mode information on the display left the 
captain without any information or reason to ques- 
tion the status of the autopilot. 

This incident clearly highlights the complexity 
and problems that can be introduced by automa- 
tion. Unfortunately, it is not a unique occurrence. 
Degani (2004) described similar accounts of diffi- 
culties encountered with other automated systems 
including cruise control in automobiles and blood 
pressure devices. 

Research on human interaction with automa- 
tion has shown that it does not always make the 
job easier. Instead, it changes the nature of work. 
More specifically, automation changes the way ac- 
tivities are distributed or carried out and can 
therefore introduce new and different types of 
problems (Woods, 1996). Automation can also 
lead to different types of errors because operator 
goals may be incongruent with the goals of sys- 
tems and subsystems (Sarter & Woods, 1995; 
Wiener, 1989). Woods (1996) argued further that 
in systems where subcomponents are tightly cou- 
pled, problems may propagate more quickly and 
be more difficult to isolate. In addition, highly au- 
tomated systems leave fewer activities for indi- 
viduals to perform. Consequently, the operator 
becomes a more passive monitor instead of an ac- 
tive participant. Parasuraman, Mouloua, Molloy, 
and Hilburn (1996) have shown that this shift 
from performing tasks to monitoring automated 
systems can actually inhibit one's ability to detect 
critical signals or warning conditions. Further, an 
operator's manual skills can begin to deteriorate in 
the presence of long periods of automation (Wick- 
ens, 1992). 



Adaptive Automation 

Given the problems associated with automation 
noted above, researchers and developers have be- 
gun to turn their attention to alternative methods 
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for implementing automated systems. Adaptive au- 
tomation is one such method that has been pro- 
posed to address some of the shortcomings of 
traditional automation. In adaptive automation, 
the level of automation or the number of systems 
operating under automation can be modified in 
real time. In addition, changes in the state of au- 
tomation can be initiated by either the human or 
the system (Hancock & Chignell, 1987; Rouse, 
1976; Scerbo, 1996). Consequently, adaptive au- 
tomation enables the level or modes of automation 
to be tied more closely to operator needs at any 
given moment (Parasuraman, Bahri, Deaton, Mor- 
rison, & Barnes, 1992). 

Adaptive automation systems can be described 
as either adaptable or adaptive. Scerbo (2001) de- 
scribed a taxonomy of adaptive technology. One 
dimension of this taxonomy concerns the underly- 
ing source of flexibility in the system, that is, 
whether the information displayed or the functions 
themselves are flexible. A second dimension ad- 
dresses how the changes are invoked. In adaptable 
systems, changes among presentation modes or in 
the allocation of functions are initiated by the user. 
By contrast, in adaptive systems both the user and 
the system can initiate changes in the state of the 
system. 

The distinction between adaptable and adap- 
tive technology can also be described with respect 
to authority and autonomy. Sheridan and Verplank 
(1978) described several levels of automation that 
range from completely manual to semiautomatic to 
fully automatic. As the level of automation in- 
creases, systems take on more authority and auton- 
omy. At the lower levels of automation, systems 
may offer suggestions to the user. The user can ei- 
ther veto or accept the suggestions and then imple- 
ment the action. At moderate levels, the system 
may have the autonomy to carry out the suggested 
actions once accepted by the user. At higher levels, 
the system may decide on a course of action, im- 
plement the decision, and merely inform the user. 
With respect to Scerbo's (2001) taxonomy, adapt- 
able systems are those in which the operator main- 
tains authority over invoking changes in the state 
of the automation (i.e., they reflect a superordinate- 
subordinate relationship between the operator and 
the system). In adaptive systems, on the other 
hand, authority over invocation is shared. Both the 
operator and the system can initiate changes in 
state of the automation. 



There has been some debate over who should 
have control over changes among modes of opera- 
tion. Some argue that operators should always have 
authority over the system because they are ulti- 
mately responsible for the behavior of the system. 
In addition, it is possible that operators may be 
more efficient at managing resources when they 
can control changes in the state of automation 
(Billings & Woods, 1994; Malin & Schreckeng- 
host, 1992). Many of these arguments are based on 
work with life-critical systems in which safe opera- 
tion is of the utmost concern. However, it is not 
clear that strict operator authority over changes 
among automation modes is always warranted. 
There may be times when the operator is not the 
best judge of when automation is needed. For ex- 
ample, changes in automation may be needed at 
the precise moment when the operator is too busy 
to make those changes (Wiener, 1989). Further, Ina- 
gaki, Takae, and Moray (1999) have shown mathe- 
matically that the best piloting decisions concerning 
whether to abort a takeoff are not those where either 
the human or the avionics maintain full control. In- 
stead, the best decisions are made when the pilot 
and the automation share control. 

Scerbo (1996) has argued that in some haz- 
ardous situations where the operator is vulnerable, 
it would be extremely important for the system to 
have authority to invoke automation. If lives are at 
stake or the system is in jeopardy, allowing the sys- 
tem to intervene and circumvent the threat or min- 
imize the potential damage would be paramount. 
For example, it is not uncommon for many of to- 
day's fighter pilots to sustain G forces high enough 
to render them unconscious for periods of up to 12 
seconds. Conditions such as these make a strong 
case for system-initiated invocation of automation. 
An example of one such adaptive automation sys- 
tem is the Ground Collision-Avoidance System 
(GCAS) developed and tested on the F-16D (Scott, 
1999). The system assesses both internal and ex- 
ternal sources of information and calculates the 
time it will take until the aircraft breaks through a 
pilot-determined minimum altitude. The system is- 
sues a warning to the pilot. If no action is taken, an 
audio "fly up" warning is then presented and the 
system takes control of the aircraft. When the sys- 
tem has maneuvered the aircraft out of the way of 
the terrain, it returns control of the aircraft to the 
pilot with the message, "You got it." The interven- 
tion is designed to right the aircraft quicker than 
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any human pilot can respond. Indeed, test pilots 
who were given the authority to override GCAS 
eventually conceded control to the adaptive system. 



Adaptive Strategies 

There are several strategies by which adaptive au- 
tomation can be implemented (Morrison & Gluck- 
man, 1994; Rouse & Rouse, 1983; Parasuraman 
et al., 1992). One set of strategies addresses system 
functionality. For instance, entire tasks can be allo- 
cated to either the system or the operator, or a spe- 
cific task can be partitioned so that the system and 
operator each share responsibility for unique por- 
tions of the task. Alternatively, a task could be 
transformed to a different format to make it easier 
(or more challenging) for the operator to perform. 
A second set of strategies concerns the trigger- 
ing mechanism for shifting among modes or levels 
of automation (Parasuraman et al., 1992; Scerbo, 
Freeman, & Mikulka, 2003). One approach relies 
on goal-based strategies. Specifically, changes 
among modes or levels of automation are triggered 
by a set of criteria or external events. Thus, the sys- 
tem might invoke the automatic mode only during 
specific tasks or if it detects an emergency situa- 
tion. Another approach would be to use real-time 
measures of operator performance to invoke the 
changes in automation. A third approach uses 
models of operator performance or workload to 
drive the adaptive logic (Hancock & Chignell, 
1987; Rouse, Geddes, & Curry, 1987-1988). For 
example, a system could estimate current and fu- 
ture states of an operator's activities, intentions, re- 
sources, and performance. Information about the 
operator, the system, and the outside world could 
then be interpreted with respect to the operator's 
goals and current actions to determine the need 
for adaptive aiding. Finally, psychophysiological 
measures that reflect operator workload can also be 
used to trigger changes among modes. 



Examples of Adaptive 
Automation Systems 

Adaptive automation has its beginnings in artificial 
intelligence. In the 1970s, efforts were directed 
toward developing adaptive aids to help allocate 
tasks between humans and computers. By the 



1980s, researchers began developing adaptive in- 
terfaces. For instance, Wilensky, Arens, and Chin 
(1984) developed the UNIX Consultant (UC) to 
provide general information about UNIX, proce- 
dural information about executing UNIX com- 
mands, and debugging information. The UC could 
analyze user queries, deduce user goals, monitor 
the user's interaction history, and present the sys- 
tem's response. 

Associate Systems 

Adaptive aiding concepts were applied in a more 
comprehensive manner in the Defense Advanced 
Research Projects Agency (DARPA) Pilot's Associate 
program (Hammer & Small, 1995). The goal of the 
program was to use intelligent systems to provide 
pilots with the appropriate information, in the 
proper format, at the right time. The Pilot's Associ- 
ate could monitor and assess the status of its own 
systems as well as events in the external environ- 
ment. The information could then be evaluated and 
presented to the pilot. The Pilot's Associate could 
also suggest actions for the pilot to take. Thus, the 
system was designed to function as an assistant for 
the pilot. 

In the 1990s, the U.S. Army attempted to take 
this associate concept further in its Rotorcraft Pi- 
lot's Associate (RPA) program. The goal was to cre- 
ate an associate that could serve as a "junior crew 
member" (Miller & Hannen, 1999). A major com- 
ponent of the RPA is the Cognitive Decision Aiding 
System (CDAS), which is responsible for detecting 
and organizing incoming data, assessing the inter- 
nal information regarding the status of the aircraft, 
assessing external information about target and 
mission status, and feeding this information into a 
series of planning and decision-making modules. 
The Cockpit Information Manager (CIM) is the 
adaptive automation system for the CDAS. The 
CIM is designed to make inferences about current 
and impending activities for the crew, allocate tasks 
among crew members as well as to the aircraft, and 
reconfigure cockpit displays to support the ability 
of the "crew-automation team" to execute those ac- 
tivities (see figure 16.1). The CIM monitors crew 
activities and external events and matches them 
against a database of tasks to generate inferences 
about crew intentions. The CIM uses this informa- 
tion to make decisions about allocating tasks, pri- 
oritizing information to be presented on limited 
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Figure 16.1. The Rotorcraft Pilot's 
Associate cockpit in a simulated 
environment. Reprinted from 
Knowledge-Based Systems, 12, Miller, 
C. A., and Hansen, M. D., The Rotor- 
craft Pilot's Associate: Design and 
evaluation of an intelligent user inter- 
face for cockpit information manage- 
ment, pp. 443-456, copyright 1999, 
with permission from Elsevier. 



display spaces, locating pop-up windows, adding 
or removing appropriate symbology from displays, 
and adjusting the amount of detail to be presented 
in displays. Perhaps most important, the CIM in- 
cludes a separate display that allows crew members 
and the system to coordinate the task allocation 
process and communicate their intentions (located 
above the center display in figure 16.1). The need 
for communication among members is strong for 
highly functioning human teams and, as it turned 
out, was essential for user acceptance of the RPA. 
Evaluations from a sample of pilots indicated that 
the RPA often provided the right information at the 
right time. Miller and Hannen reported that in the 
initial tests, no pilot chose to turn off the RPA. 

The RPA was an ambitious attempt to create an 
adaptive automation system that would function as 
a team member. Several characteristics of this effort 
are particularly noteworthy. First, a great deal of the 
intelligence inherent in the system was designed to 
anticipate user needs and be proactive about recon- 
figuring displays and allocating tasks. Second, both 
the users and the system could communicate their 
plans and intentions, thereby reducing the need to 
decipher what the system was doing and why it 
was doing it. Third, unlike many other adaptive 
automation systems, the RPA was designed to sup- 
port the simultaneous activities of multiple users. 



Although the RPA is a significant demonstra- 
tion of adaptive automation, it was not designed 
from the neuroergonomics perspective. It is true 
that a good deal of knowledge about cognitive pro- 
cessing related to decision making, information 
representation, task scheduling, and task sharing 
was needed to create the RPA, but the system was 
not built around knowledge of brain functioning. 

Brain-Based Systems 

An example of adaptive automation that follows 
the neuroergonomics approach can be found in 
systems that use psychophysiological indices to 
trigger changes in the automation. There are many 
psychophysiological indices that reflect underlying 
cognitive activity, arousal levels, and external task 
demands. Some of these include cardiovascular 
measures (e.g., heart rate, heart rate variability), 
respiration, galvanic skin response (GSR), ocular 
motor activity, and speech, as well as those that re- 
flect cortical activity such as the electroencephalo- 
gram (EEG; see also chapter 2, this volume) and 
event-related potentials (ERPs; see also chapter 3, 
this volume) derived from EEG signals to stimulus 
presentations. Additional cortical measures include 
functional magnetic resonance imaging (fMRI; see 
also chapter 4, this volume), and near-infrared 
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spectrometry (NIRS; see also chapter 5, this vol- 
ume) that measures changes in oxygenated and de- 
oxygenated hemoglobin (see Byrne & Parasuraman, 
1996, for a review of the use of psychophysiologi- 
cal measures in adaptive systems). One of the most 
important advantages to brain-based systems for 
adaptive automation is that they provide a continu- 
ous measure of activity in the presence or absence 
of overt behavioral responses (Byrne & Parasura- 
man, 1996; Scerbo et al, 2001). 

The first brain-based adaptive system was de- 
veloped by Pope, Bogart, and Bartolome (1995). 
Their system uses an index of task engagement 
based upon ratios of EEG power bands (alpha, 
beta, theta, etc.). The EEG signals are recorded from 
several locations on the scalp and are sent to a Lab- 
View Virtual Instrument that determines the power 
in each band for all recording sites and then calcu- 
lates the engagement index used to change a track- 
ing task between automatic and manual modes. 
The system recalculates the engagement index every 
2 seconds and changes the task mode if necessary. 
Pope and his colleagues studied several different 
engagement indices under both negative and posi- 
tive feedback contingencies. They argued that under 
negative feedback the system should switch modes 
more frequently in order to maintain a stable level 



of engagement. By contrast, under positive feedback 
the system should be driven to extreme levels and 
remain there longer (i.e., fewer switches between 
modes). Moreover, differences in the frequency of 
task mode switches obtained under positive and 
negative feedback conditions should provide infor- 
mation about the sensitivity of various engagement 
indices. Pope et al. found that the engagement in- 
dex based on the ratio of beta/(alpha + theta) 
proved to be the most sensitive to differences be- 
tween positive and negative feedback. 

The study by Pope et al. (1995) showed that 
their system could be used to evaluate candidate 
engagement indices. Freeman, Mikulka, Prinzel, 
and Scerbo (1999) expanded upon this work and 
studied the operation of the system in an adaptive 
context. They asked individuals to perform the 
compensatory tracking, resource management, and 
system monitoring tasks from the Multi-Task At- 
tribute Battery (MAT; Comstock & Arnegard, 
1991). Figure 16.2 shows a participant performing 
the MAT task while EEG signals are being recorded. 
In their study, all tasks remained in automatic mode 
except the tracking task, which shifted between au- 
tomatic and manual modes. They also examined 
performance under both negative and positive feed- 
back conditions. Under negative feedback, the 




Figure 16.2. An operator performing the MAT task while EEG signals are recorded. 
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tracking task was switched to or maintained in au- 
tomatic mode when the index increased above a 
preestablished baseline, reflecting high engage- 
ment. By contrast, the tracking task was switched to 
or maintained in manual mode when the index de- 
creased below the baseline, reflecting low engage- 
ment. The opposite schedule of task changes 
occurred under the positive feedback conditions. 
Freeman and his colleagues argued that if the sys- 
tem could moderate workload, better tracking per- 
formance should be observed under negative as 
compared to positive feedback conditions. Their re- 
sults confirmed this prediction. In subsequent stud- 
ies, similar findings resulted when individuals 
performed the task over much longer intervals and 
under conditions of high and low task load (see 
Scerboetal.,2003). 

More recently, St. John, Kobus, Morrison, and 
Schmorrow (2004) described a new DARPA pro- 
gram aimed at enhancing an operator's effectiveness 
by managing the presentation of information and 
cognitive processing capacity through cognitive 
augmentation derived from psychophysiological 
measures. The goal of the program is to develop 
systems that can detect an individual's cognitive 
state and then manipulate task parameters to over- 
come perceptual, attentional, and working memory 
bottlenecks. Unlike the system described by Pope 
et al. (1995) that relies on a single psychophysio- 
logical measure, EEG, the augmented cognition sys- 
tems use multiple measures including NIRS, GSR, 
body posture, and EEG. The physiological measures 
are integrated to form gauges that reflect constructs 
such as effort, arousal, attention, and workload. 
Performance thresholds are established for each 
gauge to trigger mitigation strategies for modifying 
the task. Some of these mitigation strategies include 
switching between verbal and spatial information 
formats, reprioritizing or rescheduling tasks, or 
changing the level of display detail. 

Wilson and Russell (2003, 2004) reported on 
their experiences with an augmented cognitive sys- 
tem designed to moderate workload. Operators 
were asked to perform a target identification task 
with a simulated uninhabited combat air vehicle un- 
der different levels of workload. They recorded EEG 
from six sites as well as heart, blink, and respiration 
rates. In their system, the physiological data were 
analyzed by an artificial neural network (ANN). The 
ANN was trained to distinguish between high and 
low levels of operator workload in real time. The 



output from the ANN was used to trigger changes in 
the task to moderate workload. Comparisons among 
adaptive aiding, no aid, or random aiding revealed 
some performance benefits and lower ratings of sub- 
jective workload for the adaptive aiding condition 
under the more difficult condition. 

Taken together, the findings from these studies 
suggest that it is indeed possible to obtain indices 
of one's brain activity and use that information to 
drive an adaptive automation system to improve 
performance and moderate workload. There are, 
however, still many critical conceptual and techni- 
cal issues (e.g., making the recording equipment 
less obtrusive and obtaining reliable signals in 
noisy environments) that must be overcome before 
systems such as these can move from the labora- 
tory to the field (Scerbo et al., 2001). 

Further, many issues still remain surrounding 
the sensitivity and diagnosticity of psychophysio- 
logical measures in general. There is a fundamental 
assumption that psychophysiological measures pro- 
vide a reliable and valid index of underlying con- 
structs such as arousal or attention. In addition, 
variations in task parameters that affect those con- 
structs must also be reflected in the measures 
(Scerbo et al., 2001). In fact, Veltman and Jansen 
(2004) have argued that there is no direct relation 
between information load and physiological mea- 
sures or state estimators because an increase in task 
difficulty does not necessarily result in a physiologi- 
cal response. According to their model, perceptions 
of actual performance are compared to performance 
requirements. If attempts to eliminate the difference 
between perceived and required levels of perfor- 
mance are unsuccessful, one may need to increase 
mental effort or change the task goals. Both actions 
have consequences. Investing more effort can be fa- 
tiguing and result in poorer performance. Likewise, 
changing task goals (e.g., slowing down, skipping 
low-priority tasks, etc.) can also result in poorer 
performance. They suggest that in laboratory exper- 
iments, it is not unusual for individuals to compen- 
sate for increases in demand by changing task goals 
because there are no serious consequences to this 
strategy. However, in operational environments, 
where the consequences are real and operators are 
highly motivated, changing task goals may not be 
an option. Thus, they are much more likely to in- 
vest the effort needed to meet the required levels of 
performance. Consequently, Veltman and Jansen 
contend that physiological measures can only be 
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valid and reliable in an adaptive automation envi- 
ronment if they are sensitive to information about 
task difficulty, operator output, the environmental 
context, and stressors. 

Another criticism of current brain-based adap- 
tive automation systems is that they are primarily re- 
active. Changes in external events or brain activity 
must be recorded and analyzed before any instruc- 
tions can be sent to modify the automation. All of 
this takes time and even with short delays, the sys- 
tem must still wait for a change in events to react. 
Recently, however, Forsythe, Kruse, and Schmorrow 
(2005) described a brain-based system that also in- 
corporates a cognitive model of the operator. The 
system is being developed by DaimlerChrysler 
through the DARPA Augmented Cognition program 
to support driver behavior. Information is recorded 
from the automobile (e.g., steering wheel angle, lat- 
eral acceleration) as well as the operator (e.g., head 
turning, postural adjustments, and vocalizations) 
and combined with EEG signals to generate infer- 
ences about workload levels corresponding to differ- 
ent driving situations. In this regard, the system is a 
hybrid of brain-based and operator modeling ap- 
proaches to adaptive automation and can be more 
proactive than current adaptive systems that rely 
solely on psychophysiological measures. 



Workload and Situation Awareness 

Workload 

One of the arguments for developing adaptive au- 
tomation is that this approach can moderate opera- 
tor workload. Most of the research to date has 
assessed workload through primary task perfor- 
mance or physiological indices (see above). Kaber 
and Riley (1999), however, conducted an experi- 
ment using both primary and secondary task mea- 
sures. They had their participants perform a 
simulated radar task where the object was to elimi- 
nate targets before they reached the center of the 
display or collided with one another. During man- 
ual control, the participants were required to assess 
the situation on the display, make decisions about 
which targets to eliminate, and implement those 
decisions. During a shared condition, the partici- 
pant and the computer could each perform the sit- 
uation assessment task. The computer scheduled 
and implemented the actions, but the operator had 



the ability to override the computer's plans. The 
participants were also asked to perform a second- 
ary task requiring them to monitor the movements 
of a pointer and correct any deviations outside of 
an ideal range. Performance on the secondary task 
was used to invoke the automation on the primary 
task. For half of the participants, the computer 
suggested changes between automatic or manual 
operation of the primary task, and for the remain- 
ing participants, those changes were mandated. 

Kaber and Riley (1999) found that shared con- 
trol resulted in better performance than manual 
control on the primary task. However, the results 
showed that mandating the use of automation also 
bolstered performance during periods of manual 
operation. Regarding the secondary task, when use 
of automation was mandated, workload was lower 
during periods of automation; however, under pe- 
riods of manual control, workload levels actually 
increased and were similar to those seen when its 
use was suggested. These results show that authority 
over invoking changes between modes had differen- 
tial effects on workload during periods of manual 
and automated operation. Specifically, Kaber and 
Riley found that the requirement to consider com- 
puter suggestions to invoke automation led to higher 
levels of workload during periods of shared or auto- 
mated control than when those decisions were dic- 
tated by the computer. 

Situation Awareness 

Thus far, there have been few attempts to study the 
effects of adaptive automation on situation aware- 
ness (SA). Endsley (1995) described SA as the abil- 
ity to perceive elements in the environment, 
understand their meaning, and make projections 
about their status in the near future. One might 
assume that efforts to moderate workload through 
adaptive automation would lead to enhanced SA; 
however, that relationship has yet to be demon- 
strated empirically. In fact, within an adaptive par- 
adigm, periods of high automation could lead to 
poor SA and make returning to manual operations 
more difficult. The findings of Kaber and Riley 
(1999) regarding secondary task performance de- 
scribed above support this notion. 

Bailey, Scerbo, Freeman, Mikulka, and Scott 
(2003) examined the effects of a brain-based adap- 
tive automation system on SA. The participants were 
given a self-assessment measure of complacency 
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toward automation (i.e., the propensity to become 
reliant on automation; see Singh, Molloy, & Para- 
suraman, 1993) and separated into groups who 
scored either high or low on the measure. The par- 
ticipants performed a modified version of the MAT 
battery that included a number of digital and ana- 
log displays (e.g., vertical speed indicator, GPS 
heading, oil pressure, and autopilot on/off) used to 
assess SA. Participants were asked to perform the 
compensatory tracking task during manual mode 
and to monitor that display during automatic 
mode. Half of the participants in each complacency 
potential group were assigned to either an adaptive 
or yoke control condition. In the adaptive condi- 
tion, Bailey et al. used the system modified by 
Freeman et al. (1999) to derive an EEG-based en- 
gagement index to control the task mode switches. 
In the other condition, each participant was yoked 
to one of the individuals in the adaptive condition 
and received the same pattern of task mode 
switches; however, their own EEG had no effect on 
system operation. All participants performed three 
15-minute trials. At the end of each trial, the com- 
puter monitor went blank and the experimenter 
asked the participants to report the current values 
for a sample of five displays. Participants' reports 
for each display were then compared to the actual 
values to provide a measure of SA (Endsley 2000). 
Bailey and his colleagues (2003) found that the 
effects of the adaptive and yoke conditions were 
moderated by complacency potential. Specifically, 
for individuals in the yoke control conditions, 
those who were high as compared to low in com- 
placency potential had much lower levels of SA. 
On the other hand, there was no difference in SA 
scores for high- and low-complacency individuals 
in the adaptive conditions. More important, the SA 
scores for both high- and low-complacency indi- 
viduals were significantly higher than those of the 
low-complacency participants in the yoke control 
condition. The authors argued that a brain-based 
adaptive automation system could ameliorate the 
effects of complacency by increasing available at- 
tentional capacity and in turn, improving SA. 



Human-Computer Etiquette 

Interest has been shown in the merits of an eti- 
quette for human-computer interaction. Miller 
(2002) described etiquette as a set of prescribed 



and proscribed behaviors that permit meaning and 
intent to be ascribed to actions. Etiquette serves to 
make social interactions more cooperative and po- 
lite. Importantly, rules of etiquette allow one to 
form expectations regarding the behaviors of oth- 
ers. In fact, Nass, Moon, and Carney (1999) have 
shown that people adopt many of the same social 
conventions used in human-human interactions 
when they interact with computers. Moreover, they 
also expect computers to adhere to those same 
conventions when computers interact with users. 

Miller (2004) argued that when humans inter- 
act with systems that incorporate intelligent agents, 
they may expect those agents to conform to ac- 
cepted rules of etiquette. However, the norms may 
be implicit and contextually dependent: What is 
acceptable for one application may violate expecta- 
tions in another. Thus, there may be a need to un- 
derstand the rules under which computers should 
behave and be more polite. 

Miller (2004) also claimed that users ascribe 
expectations regarding human etiquette to their 
interactions with adaptive automation. In their 
work with the RPA, Miller and Hannen (1999) ob- 
served that much of the dialogue between team 
members in a two-seat aircraft was focused on 
communicating plans and intentions. They rea- 
soned that any automated assistant would need to 
communicate in a similar manner to be accepted 
as a team player. Consequently, the CIM described 
earlier was designed to allow users and the system to 
communicate in a conventionally accepted manner. 

The benefits of adopting a human-computer 
etiquette are described by Parasuraman and Miller 
(2004) in a study of human-automation interac- 
tions. In particular, they focused on interruptions. 
In their study, participants were asked to perform 
the tracking and fuel resource management tasks 
from the MAT battery. A third task required partici- 
pants to interact with an automated system that 
monitored engine parameters, detected potential 
failures, and offered advice on how to diagnose 
faults. The automation support was implemented 
in two ways. Under the "patient" condition, the au- 
tomated system would withhold advice if the user 
was in the act of diagnosing the engines, or provide 
a warning, wait 5 seconds, and then offer advice if 
it determined the user was not interacting with the 
engines. By contrast, under the "impatient" condi- 
tion the automated system offered its advice with- 
out warning while the user was performing the 
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diagnosis. Parasuraman and Miller referred to the 
patient and impatient automation as examples of 
good and poor etiquette, respectively. In addition, 
they examined two levels of system reliability. Un- 
der low and high reliability, the advice was correct 
60% and 80% of the time, respectively. 

As might be expected, performance was better 
under high as opposed to low reliability. Further, 
Parasuraman and Miller (2004) found that when 
the automated system functioned under the good 
etiquette condition, operators were better able to 
diagnose engine faults regardless of reliability level. 
In addition, overall levels of trust in the automated 
system were much higher under good etiquette 
within the same reliability conditions. Thus, "rude" 
behavior made the system seem less trustworthy ir- 
respective of reliability level. Several participants 
commented that they disliked being interrupted. 
The authors argued that systems designed to con- 
form to rules of etiquette may enhance performance 
beyond what might be expected from system relia- 
bility and may even compensate for lower levels of 
reliability. 

Parasuraman and Miller's (2004) findings were 
obtained with a high-criticality simulated system; 
however, the rules of etiquette (or interruptions) 
may be equally important for business or home ap- 
plications. Bubb-Lewis and Scerbo (2002) exam- 
ined the effects of different levels of communication 
on task performance with a simulated adaptive 
interface. Specifically, participants worked with a 
computer "partner" to solve problems (e.g., deter- 
mining the shortest mileage between two cites or 
estimating gasoline consumption for a trip) using a 
commercial travel-planning software package. In 
their study, the computer partner was actually a 
confederate in another room who followed a strict 
set of rules regarding how and when to intervene 
to help complete a task for the participant. In addi- 
tion, they studied four different modes of commu- 
nication that differed in the level of restriction 
ranging from context-sensitive natural language to 
no communication at all. The results showed that 
as restrictions on communication increased, partic- 
ipants were less able to complete their tasks, which 
in turn caused the computer to intervene more 
often to complete the tasks. This increase in inter- 
ventions also led the participants to rate their inter- 
actions with the computer partner more negatively. 
Thus, these findings suggest that even for less criti- 
cal systems, poor etiquette makes a poor impres- 



sion. Apparently, no one likes a show-off, even if it 
is the computer. 



Living with Adaptive Automation 

Adaptive automation is also beginning to find its 
way into commercial and more common technolo- 
gies. Some examples include the adaptive cruise 
control found on several high-end automobiles and 
"smart homes" that control electrical and heating 
systems to conform to user preferences. 

Mozer (2004) has described his experiences liv- 
ing in an adaptive home of his own creation. The 
home was designed to regulate air and water tem- 
perature and lighting. The automation monitors the 
inhabitants activities and makes inferences about 
the inhabitants behavior, predicts future needs, and 
adjusts the temperature or lighting accordingly. 
When the automation fails to meet the user's expec- 
tations, the user can set the controls manually. 

The heart of the adaptive home is the adaptive 
control of home environment (ACHE), which func- 
tions to balance two goals: user desires and energy 
conservation. Because these two goals can conflict 
with one another, the system uses a reinforcement 
learning algorithm to establish an optimal control 
policy. With respect to lighting, the ACHE controls 
multiple independent light fixtures, each with mul- 
tiple levels of intensity (see figure 16.3). The ACHE 
encompasses a learning controller that selects light 
settings based on current states. The controller re- 
ceives information about an event change that is 
moderated by a cost evaluator. A state estimator 
generates high-level information about inhabitant 
patterns and integrates it with output from an oc- 
cupancy model as well as information regarding 
levels of natural light available to make decisions 
about changes in the control settings. The state es- 
timator also receives input from an anticipator 
module that uses neural nets to predict which 
zones are likely to be inhabited within the next 2 
seconds. Thus, if the inhabitant is moving within 
the home, the ACHE can anticipate the route and 
adjust the lights before he arrives at his destination. 
Mozer (2004) recorded the energy costs as well as 
costs of discomfort (i.e., incorrect predictions and 
control settings) for a month and found that both 
decreased and converged within about 24 days. 

Mozer (2004) had some intriguing observations 
about his experiences living in the adaptive house. 
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Figure 16.3. Michael Mozer's adaptive house. An interior photo of the great room is shown on the left. On the 
right is a photo of the data collection room where sensor information terminates in a telephone punch panel and 
is routed to a PC. A speaker control board and a microcontroller for the lights, electric outlets, and fans are also 
shown here. (Photos courtesy of Michael Mozer). 



First, he found that he generated a mental model of 
the ACHE's model of his activities. Thus, he knew 
that if he were to work late at the office, the house 
would be "expecting" him home at the usual time, 
and he often felt compelled to return home. Further, 
he admitted that he made a conscious effort to be 
more consistent in his activities. He developed a 
meta-awareness of his occupancy patterns and rec- 
ognized that as he made his behavior more regular, it 
facilitated the operation of the ACHE, which in turn 
helped it to save energy and maximize his comfort. 
In fact, Mozer claimed, "the ACHE trains the inhabi- 
tant, just as the inhabitant trains the ACHE" (p. 293). 
Mozer (2004) also discovered the value of com- 
munication. At one point, he noticed a bug in the 
hardware and modified the system to broadcast a 
warning message throughout the house to reset the 
system. After the hardware problem had been ad- 
dressed, however, he retained the warning message 
because it provided useful information about how 
his time was being spent. He argued that there were 
other situations where the user could benefit from 
being told about consequences of manual overrides. 



Conclusion 

The development of adaptive automation repre- 
sents a qualitative leap in the evolution of technol- 



ogy. Users of adaptive automation will be faced 
with systems that differ significantly from the auto- 
mated technology of today. These systems will be 
much more complex from both the users' and de- 
signers' perspectives. Scerbo (1996) argued that 
adaptive automation systems will need time to 
learn about users and users will need time to un- 
derstand the automation. In Mozer's (2004) case, 
he and his home needed almost a month to adjust 
to one another. Further, users may find that adap- 
tive systems are less predictable due to the variabil- 
ity and inconsistencies of their own behavior. 
Thus, users are less likely to think of these systems 
as tools, machines, or even traditional computer 
programs. As Mozer indicated, he soon began to 
think about how his adaptive home would respond 
to his behavior. Others have suggested that interact- 
ing with adaptive systems is more like interacting 
with a teammate or coworker (Hammer & Small, 
1995; Miller & Hannen, 1999; Scerbo, 1994). 

The challenges facing designers of adaptive sys- 
tems are significant. Current methods in system 
analysis, design, and evaluation fall short of what is 
needed to create systems that have the authority 
and autonomy to swap tasks and information with 
their users. These systems require developers to be 
knowledgeable about task sharing, methods for 
communicating goals and intentions, and even as- 
sessment of operator states of mind. In fact, Scerbo 
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(1996) has argued that researchers and designers of 
adaptive technology need to understand the social, 
organizational, and personality issues that impact 
communication and teamwork among humans 
to create more effective adaptive systems. In this 
regard, Miller's (2004) ideas regarding human- 
computer etiquette may be paramount to the de- 
velopment of successful adaptive systems. 

Thus far, most of the adaptive automation sys- 
tems that have been developed address life-critical 
activities where the key concerns surround the 
safety of the operator, the system itself, and recipi- 
ents of the system's services. However, the technol- 
ogy has also been applied in other contexts where 
the consequences of human error are less severe 
(e.g., Mozer's adaptive house). Other potential ap- 
plications might include a personal assistant, butler, 
tutor, secretary, or receptionist. Moreover, adaptive 
automation could be particularly useful when in- 
corporated into systems aimed at training and skill 
development as well as entertainment. 

To date, most of the adaptive automation sys- 
tems that have been developed were designed to 
maximize the user-system performance of a single 
user. Thus, they are user independent (i.e., designed 
to improve the performance of any operator). How- 
ever, overall user-system performance is likely to be 
improved further if the system is capable of learning 
and adjusting to the behavioral patterns of its user, 
as was shown by Mozer (2004). This would indeed 
make the line between human and machine more 
flimsy, as Clark (2003) has suggested. Although 
building systems capable of becoming more user 
specific might seem like a logical next step, that ap- 
proach would introduce a new and significant chal- 
lenge for designers of adaptive automation — 
addressing the unique needs of multiple users. The 
ability of Mozer's house to successfully adapt to his 
routines is due in large part to his being the only in- 
habitant. One can imagine the challenge faced by an 
adaptive system trying to accommodate the wishes 
of two people who want the temperature set at dif- 
ferent levels. 

The problem of accommodating multiple users 
is not unique to adaptive automation. In fact, the 
challenge arises from a fundamental aspect of hu- 
manity. People are social creatures and, as such, they 
work in teams, groups, and organizations. Moreover, 
they can be colocated or distributed around the 
world and networked together. Developers of collab- 
orative meeting and engineering software realize 



that one cannot optimize the individual human- 
computer interface at the expense of interfaces that 
support team and collaborative activities. Conse- 
quently, even systems designed to work more effi- 
ciently based on knowledge of brain functions must 
ultimately take into consideration groups of people. 
Thus, the next great challenge for the neuroer- 
gonomics approach may lie with an understanding 
of how brain activity of multiple operators in social 
situations can improve the organizational work envi- 
ronment. 



Main Points 

1. One source of interest in neuroergonomics 
stems from research showing that automated 
systems do not always reduce workload and 
create safer working conditions. 

2. Adaptive automation refers to systems in 
which the level of automation or the number 
of subsystems operating under automation can 
be modified in real time. In addition, changes 
in the state of automation can be initiated by 
either the operator or the system. 

3. Adaptive automation has traditionally been 
implemented in associate systems that use 
models of operator behavior and workload, 
but recent brain-based systems that follow the 
neuroergonomics approach show much 
promise. 

4. User interactions with adaptive automation 
may be improved through an understanding of 
human-computer etiquette. 

5. Working with adaptive automation can be like 
working with a teammate. Further, living with 
adaptive automation can modify both the 
system and the user's behavior. 

6. The next great challenge for neuroergonomics 
may be to address the individual needs of 
multiple users. 
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Virtual Reality and 
Neuroergonomics 



Virtual reality (VR) describes the use of computer- 
generated stimuli and interactive devices to situate 
participants in simulated surroundings that resem- 
ble real or fantasy worlds. As early as 1965, Ivan 
Sutherland foresaw the potential of computer 
graphics to create a window into a virtual world. In 
its very first episode, the late 1980s television show 
Star Trek: The Next Generation introduced the con- 
cept of a virtual environment into the public imag- 
ination with the spaceship's "holodeck," which 
persists today as the archetype for a virtual world. 
The usage of the term virtual reality is attributed to 
Jaron Lanier, who defined it as "a computer- 
generated, interactive, three-dimensional environ- 
ment in which a person is immersed" (Tsirliganis, 
2001). Brooks (1999) defined a VR experience as 
"any in which the user is effectively immersed in a 
responsive virtual world." The Lanier and Brooks 
definitions both identify the two elements most 
commonly associated with VR: immersion and in- 
teraction. 

With VR, a computer-controlled setup creates 
an illusory environment that relies on visual, audi- 
tory somatosensory (e.g., haptic [tactile], vibratory, 
and proprioceptive), and even olfactory displays. 
The term presence describes the degree to which a 
person feels a part of, or engaged in, the VR world 



(figure 17.1). It is sometimes described as a sense 
of "being there." Users may navigate within the vir- 
tual world and interact with objects, obstacles, and 
inhabitants such as avatars, the digital representa- 
tions of real humans in virtual worlds. Optimally, 
participants will lose awareness of their physical 
surroundings and any sense of artificiality and will 
feel and behave in a virtual environment (VE) as 
they would in the real world. This allows presenta- 
tion of useful tasks or diversions in many circum- 
stances for entertaining, training, testing, or treating 
normal or impaired users. 

Ellis (1991, 1993, 1995), Rheingold (1991), 
and Heim (1993) have reviewed the philosophical 
underpinnings, history, and early implementations 
of VR. Brooks (1999) reviewed the state of the art 
for VR through 1999. Continued gains in com- 
puter power and display technologies since then 
have made VR better, more economical, and acces- 
sible to a wider range of users and disciplines. 
Availability of high-quality personal computer (PC) 
graphics-rendering hardware (largely due to the 
commercial demands of computer gaming applica- 
tions) and steady improvements in the quality and 
cost of video projectors (driven by presentation ap- 
plications and, more recently, by home entertain- 
ment) have dramatically reduced the cost of VR 
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Figure 1 7.1 . Immersion in virtual reality (VR). A VR user in a head-mounted display 
and harness is swimming across the Pacific. The lack of wetness or fatigue, the harness, 
and noisy spectators spoil the immersion. 



equipment. Coupled with technical advancements 
in graphics and sound rendering, gesture and 
speech recognition, modeling of biomechanics, 
motion tracking, motion platforms, force feedback 
devices, and improvements in software for creating 
complex VE databases, it is now possible to pursue 
research using VR with a modest budget and lim- 
ited technical expertise. 

It is difficult to precisely define what it takes to 
make a simulated experience VR. It is generally 
accepted that VR refers to 3-D environments in 
which participants perceive that they are situated 
and can interact with the environment in some 
way, typically by navigating through it or manipu- 
lating objects in it. The sensory experience should 
be immersive in some respects, and participants 
should have a feeling of presence. The factors that 
influence the degree and cognitive underpinnings 
of presence are under investigation (Meehan, In- 
sko, Whitton, & Brooks, 2002; Sanchez-Vives & 
Slater, 2005). Panoramic movies are typically not 
considered VR (and neither is watching "reality 
TV") because the participants are not able to inter- 
act with the objects or entities in the environment. 

VR is highly relevant to neuroergonomics be- 
cause VR can replicate situations with greater con- 
trol than is possible in the real world, allowing 



behavioral and neurophysiological observations of 
the mind and brain at work in a wide range of situ- 
ations that are impractical or impossible to observe 
in the real world. VR can be used to study the per- 
formance of hazardous tasks without putting sub- 
jects at risk of injury. For example, VR has been 
used to study the influence of disease, drugs, and 
disabilities on driving, to understand how the in- 
troduction of in-vehicle technologies such as cell 
phones and heads-up displays influence crash risk, 
to investigate underlying causes of bicycle crashes 
(Plumert, Kearney, & Cremer, 2004), and to exam- 
ine how to reduce the risk of falling from roofs and 
scaffolding (Simeonov, Hsiao, Dotson, & Ammons, 
2005). 

VR can also help train students in areas where 
novice misjudgments and errors can have devastat- 
ing outcomes, such as learning to do medical pro- 
cedures, to fly an airplane, or to operate heavy 
machinery. VR can provide a cost-effective means to 
train users in tasks that are by their nature destruc- 
tive, such as learning to fire shoulder-held missiles. 
VR can provide training for environments that are 
dangerous or inaccessible, such as mission rehearsal 
for fire training (St. Julien & Shaw, 2003), mining 
(Foster & Burton, 2003), or hostage rescue. VR is 
also proving to be useful for therapy and rehabilita- 
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tion of persons with motor, cognitive, or psychiatric 
impairments. VR is especially useful when the job 
requires spatial awareness, complex motor skills, or 
decision making to choose among possible actions 
appropriate to changing circumstances. 

As we shall see, detailed observations in these 
immersive VR environments offer a window to 
mechanisms of the human brain and cognition and 
have applications to public health and safety, educa- 
tion, training, communication, entertainment, sci- 
ence, engineering, psychology, health care, and the 
military (e.g., Barrett, Eubank, & Smith, 2005; 
Brown, 2005; Thacker, 2003). 



The Physiology of a VR Experience 

VR setups provide a closely controlled environment 
for examining links between human physiology and 
behavior. Relevant measurements in different VR 
tasks can assess body temperature, heart rate, respi- 
ratory rate, galvanic skin response, electroencephalo- 
graphic (EEG) activity, electromyographic activity, 
eyelid closure (as an index of alertness/arousal), eye 
movements (as an index of cognitive processing), 
and even brain metabolic activity. 

Physiological sensors can be fastened to a sub- 
ject or positioned remotely to record physiological 
data from a subject immersed in the VE. These in- 
struments differ on spatial and timing accuracy, 
ease and reliability of calibration, ability to accom- 
modate appliances (such as spectacles in the case 
of eye-recording systems), susceptibility to vibra- 
tion and lighting conditions, variability and re- 
liability, ability to connect with auxiliary devices, 
synchronization with the performance measures 
collected, and the ability to automate analysis 
of large data files and to visualize and analyze 
these data in commercial software and statistical 
packages. 

Physiological measurements are generally eas- 
ier to make in VR than in the real world where 
movement, lighting, and other artifacts can be a 
nuisance and a source of confounding variables. 
Synchronous measures of operator physiology and 
performance in VR scenarios can illuminate rela- 
tionships between cognition and performance in 
operators with differing levels of self-awareness 
(e.g., drivers, pilots, factory workers, medical per- 
sonnel, patients) under effects of fatigue, distrac- 
tion, drugs, or brain disorders. Some physiological 



indices that can be measured in VE are listed in 
table 17.1. 

Common technical difficulties facing re- 
searchers who record physiological measures in- 
clude how specific measures are analyzed and 
reported. Common problems include the spatial 
and timing precision of the physiological recording 
hardware, postprocessing of recorded measure- 
ments to find physiologically meaningful events 
(e.g., localization of fixation points from raw eye 
movement data or identification of microsleeps 
from EEG recordings), and synchronization of the 
stream of data on subject performance with the 
timing of simulation events and activities. Mea- 
surement devices such as those that record galvanic 
skin response or eye movements are notoriously 
temperamental and require frequent calibration, 
and dependent measures must be clearly defined 
(see figure 17.2). 

The Honeywell Laboratories AugCog team de- 
veloped a closed-loop, adaptive system that monitors 
neurophysiological measures — EEG, pupilometric, 
and physiological measures such as electrooculog- 
raphy and electrocariography — as a comprehensive 
set of cognitive gauges for assessing the cognitive 
state of a subject. The Closed-Loop Integrated Proto- 
type of a military communications scheduler is de- 
signed to schedule a soldiers tasks, communications 
rates, and modalities to prevent cognitive overload 
and provide higher situational awareness and mes- 
sage comprehension in high-workload conditions 
(Dorneich, Whitlow, Ververs, Carciofmi, & Creaser, 
2003). 

The HASTE (Human Machine Interface and 
the Safety of Traffic in Europe) group proposed 
several physiological indices to assess user work- 
load in VR for driving. Mandatory workload mea- 
sures (which had to be collected at all sites) included 
a rating scale of mental effort, eye glance measures 
(glance frequency, glance duration), measures of ve- 
hicle control while using vehicle information system 
performance, and a measure of situation awareness. 
Optional workload measures (which could be col- 
lected at sites with appropriate facilities) included 
heart rate and heart rate variability, respiration, and 
skin conductance. 

Note that physiological indices can be used to 
make inferences about cognitive and emotional ac- 
tivity during VR tasks. Rainville, Bechara, Naqvi, 
and Damasio (2006) investigated profiles of car- 
diorespiratory activity during the experience of 
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Table 17.1. Physiological Measures in Virtual Environments 



Tesl 



Description 



Electroencephalography (EEG) 

Heart rate 
Respiratory frequency 

Near-infrared spectroscopy 
Eye movements 



Eye blinks 



Cervical paraspinal 
electromyographic recordings 
Electrogastrogram 



Skin electrical resistance: 
galvanic skin response, a.k.a. 
electrodermal response 



Routinely used to determine sleep onset and microsleeps (Harrison & Home, 1996; 

Risser et af, 2000). EEG power spectral analysis may detect subtle shifts into lower 

frequency bands (theta, delta) that would be associated with developing drowsiness 

(Home & Reyner, 1996; Kecklund & Akerstedt, 1993; see also chapter 2, this 

volume). 

Variability can be assessed by subjecting continuous EKG data and may be more 

sensitive than absolute heart rate in studies of fatigued operators (Egelund, 1982). 

Changes in respirations/minute and relative amplitude of abdominal respiratory 

effort can continuously be recorded. Sleep onset is typically associated with 

decreasing abdominal respiratory effort, while thoracic effort remains stable (Ogilvie 

etal, 1989). 

Has the potential to passively monitor the effects of cognition on oxygenation of 

blood in the brain during challenges posed in VR environments. Near-infrared light 

incident on the skin diffuses through the skull into the brain and then diffuses back 

out to the skin where it can be detected as an index of brain activity (see also 

chapter 2, this volume). 

Can index attention and cognition in virtual environments. They can provide an 

index of information processing and depend on the stimulus, context for search, and 

scenario. For example, in driving on straight roads in low traffic, drivers tend to fixate 

around the focus of expansion, in the direction of forward travel in virtual 

environments. Experienced drivers may fixate farther away from the car than novices 

do (e.g., Mourant & Rockwell, 1972). Cognitive load affects eye movements. Eye 

movements generally precede driver actions over vehicle controls, so prediction of 

driver behavior may be possible. For example, driver eye and head checks may 

precede lane position and steering changes. Relevant analyses can assess fixation 

duration, distance, location (in regions of interest in the virtual environment), scan 

path length, and transitions between fixations (see chapter 7). Eye movements can 

also provide an index of drowsiness during VR (Wierwille et al., 1995; Hakkanen 

et al., 1999; Dinges & Mallis, 2000; see also chapter 7, this volume) 

Frequency is reported in blinks/minute. Blink duration is expressed as PERCLOS 

(PERcent eyelid CLOSure) in consecutive, 3-minute epochs. PERCLOS can predict 

performance degradation in studies of sleep-deprived individuals (Dinges & Mallis, 

2000). 

Reduction in activity has correlated with poor performance in fatigued operators 

(Dureman & Boden, 1972). 

Can be recorded from electrodes placed over the abdomen. Tachygastria (acute 

increased stomach activity of 4-9 cycles per minute) correlates with feelings of 

nausea and excitement. 

Increases with drowsiness (Dureman & Boden, 1972; Johns et al., 1969) and peaks 
in response to an emotionally charged stimulus (see figure 17.4). 



fear, anger, sadness, and happiness. They recorded 
electrocardiographic and respiratory activity in 43 
volunteers during recall and experiential reliving of 
potent emotional episodes or during a neutral 
episode. Multiple linear and spectral indices of 
cardiorespiratory activity were reduced to five phys- 
iologically meaningful factors using principal com- 
ponent analysis. Multivariate analyses of variance 
and effect size estimates calculated based on those 
factors confirmed the differences between the four 



emotion conditions. The results are compatible with 
the proposal of William James (J894) that afferent 
signals felt by the viscera are essential for the unique 
experience associated with distinct emotions. 

Physiological measures can provide a quantita- 
tive index of a users feeling of presence in VR. 
Meehan et al. (2002) examined reactions to stress- 
ful situations presented in VR by observing physio- 
logical measures that correlate with stress in real 
environments. They found that heart rate and skin 
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Figure 1 7.2. Physiology in virtual reality Physiology in virtual reality Top. Quad view (Top) shows a 
surprised driver (upper left panel) who is braking (lower left panel) in response to an e-dog running 
across the driver's path (lower right panel) in a virtual driving environment. Bottom. Galvanic skin 
response activity (GSR) accompanies the surprised driver's response. 



258 Technology Applications 



Figure 1 7.3. Height effects in real and 
virtual reality environments. Virtual real- 
ity environments have been used to 
study height effects and spatial cognition 
in neuroergonomic research aimed at re- 
ducing falls and injuries in work envi- 
ronments. Simeonov et al. (2005) 
compared psychological and physiologi- 
cal responses to height in corresponding 
real and virtual environments. (Courtesy 
of NIOSH, Division of Safety Research.) 
See also color insert. 




conductance significantly increased and skin tem- 
perature significantly decreased when subjects were 
exposed to a virtual pit — a room with a large hole 
in the floor that dropped 20 feet to the ground be- 
low. The most consistent correlate was heart rate. 
Simeonov et al. (2005) compared psychological 
and physiological responses to height in correspon- 
ding real and virtual environments, pictured in fig- 
ure 17.3. They found similar levels of anxiety and 
perceived risk of falling in the real and virtual envi- 
ronments. Subjects had similar skin conductance 
responses and comparable postural instabilities at 
real and virtual heights. However, while heart rates 
were elevated in real environments, heart rates were 
not stable across heights in the virtual environment. 
From a cognitive neuroscience perspective, 
Sanchez-Vives and Slater (2005) noted that VEs 
can break the everyday connection between where 
our senses tell us we are and where we actually are 
located and whom we are with. They argue that 
studies of presence, the phenomenon of behaving 
and feeling as if we are in the virtual world created 
by computer displays, may aid the study of percep- 
tion and consciousness. 



Augmented Reality 

The term augmented reality (AR) refers to the com- 
bining of real and artificial stimuli, generally with 
the aim of improving human performance and cre- 
ativity. This typically involves overlaying computer- 
generated graphics on a stream of video images so 
that these virtual objects appear to be embedded in 
the real world. Sounds, somatosensory cues, and 
even olfactory cues can also be added to the sen- 
sory stream from the real world. The augmentation 
may highlight important objects or regions, super- 
impose informative annotations, or supplement a 
real environment. 

AR has the potential to help normal individuals 
accomplish difficult tasks by enhancing their nor- 
mal perceptual abilities with information that is 
typically unavailable. Perhaps the most visible and 
commercially successful example of AR is the yel- 
low first-down line shown on television broadcasts 
of college and professional football games. Using 
precise information on camera position and orien- 
tation and a geometric model of the shape of the 
football field, an imaginary yellow line is electroni- 
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Figure 17.4. Augmented cognition. The Rockwell Collins Synthetic Vision System overlays 
wire-frame terrain imagery rendered from an onboard database onto a heads-up display along 
with guidance symbology This improves the pilot's awareness of terrain under poor visibility 
conditions without obstructing the real- world view as visibility improves (courtesy of Rock- 
well Collins, Inc.). 



cally superimposed on the image of the football 
field to indicate the first-down line. 

Imagine a surgeon, about to begin an incision, 
who can see a patient's internal organs through that 
patient's skin, or a landscape architect who can see 
property lines overlaid on the ground and the un- 
derground pipes and conduits (Spohrer, 1999). 
Such AR applications can use semitransparent 
head-mounted displays (HMDs) to superimpose 
computer-generated images over the images of the 
real world. They could help aircraft pilots maintain 
situational awareness of weather, other air traffic, 
aircraft state, and tactical operations using an HMD 
that enhances salient features (e.g., the horizon, oc- 
cluded runway markings) or attaches labels to those 
features to identify runways, buildings, or nearby air- 
craft (http://hci.rsc.rockwell.com/AugmentedReality) . 
Such augmented information could also be pre- 
sented using a "heads-up display" located or pro- 
jected on a window along the pilot or driver's line 
of sight to the external world (see figure 17.4). 

AR might also help a cognitively impaired 
older walker to navigate better by superimposing 
visual labels for landmarks and turning points on 
images of the terrain viewed through instrumented 
spectacles. Sensors on the navigator would trans- 
mit personal position, orientation, and movements 



that enable accurate AR overlays (see chapter 8, 
this volume). 

The key challenges in creating effective AR sys- 
tems are (1) modeling the virtual objects to be em- 
bedded in the image, (2) precisely registering the 
real and virtual coordinate systems, (3) producing 
realistic lighting and shading, (4) generating im- 
ages quickly enough to avoid disconcerting lag 
when there is relative movement, and (5) building 
portable devices that do not encumber the wearer. 



Augmented Cognition 

The information presented to the user using an AR 
setup can also be used in systems for augmented 
cognition (AC). AC research aims to enhance user 
performance and cognitive capabilities through 
adaptive assistance. AC systems can employ physio- 
logical sensors to monitor a user's performance and 
regulate the information presented to the user to 
minimize stress, fatigue, and information overload. 
When an AC system identifies a state of cognitive 
overload, it modifies the presentation or pace of in- 
formation for the user, offloads information, or alerts 
others about the state of the user (Domeich, Whit- 
low, Ververs, Rogers., 2003). 
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Honeywell used a gaming environment to test 
AC for subjects engaged in a military scenario. 
Subjects played the role of a platoon leader charged 
with leading soldiers to an objective in a hostile ur- 
ban setting, shooting enemy soldiers and not their 
own (Dorneich, Whitlow, Ververs, Carciofini, & 
Creaser, 2004). Participants also received commu- 
nications, some of which required a response. This 
included status and mission updates and requests 
for information and reports. This exercise showed 
that physiological sensors could provide inputs to 
regulate the information presented so that the user 
is not overwhelmed with data or inappropriately 
distracted when full attention should be directed to 
the matters at hand. 

Finally, note that AC might also be achieved by 
very invasive means, such as implantable brain de- 
vices to enhance memory or visual processing (see 
chapter 21, this volume). 



Fidelity 

In VR, fidelity refers to the accuracy with which real 
sensory cues are reproduced. Visual cue reproduc- 
tion is crucial; Ackerman (1990) highlighted the 
key role of somatosensory information in human 
experience in her book A Natural History of the 
Senses, and Rheingold (1991) underscored its com- 
pelling importance in VR. Yet, despite relentless 
gains in image generation technology, display reso- 
lution (measured by numbers of pixels, levels of in- 
tensity, and range of colors), and processing power 
of real-time graphics rendering hardware and soft- 
ware, VR visuals still fall short of the resolution of 
the human visual system. Three-dimensional spatial 
sound, motion generation, and haptic (tactile) in- 
terfaces in VR are likewise imperfect. 

The fidelity of a computer-generated stimulus 
depends on the entire process of stimulus genera- 
tion. This includes the fidelity with which the ren- 
dering computations capture the properties of the 
physical stimulus. At present, images can be gener- 
ated in real time for only simple lighting models. 
Diffuse illumination, reflection of light between 
objects, and caustic effects (caused by uneven re- 
fraction of light) that are now commonplace in 
computer animation cannot be computed at rates 
necessary for real-time applications. As a result, 
VR images typically appear cartoonish and flat 
compared to real images, computer-generated still 



images, and Hollywood production animations, 
which can require hours of computation to render 
a single image. 

Because fidelity costs money (and money is 
scarce), researchers and practitioners must often 
make difficult choices on how to allocate equip- 
ment budgets and trade off fidelity in one area for 
another. For example, stereo imaging doubles the 
requirement for image generation because two im- 
ages must be rendered (one for each eye). Other 
systems, which use liquid crystal shutter glasses to 
alternately expose the two eyes to a single display, 
require projectors with rapid refresh rates and can 
greatly increase projector cost. Stereo imaging has 
been identified as an important factor in near-field 
perception of 3-D, particularly forjudging depth of 
objects within arm's reach. However, for applica- 
tions where the user is principally focused on objects 
at a distance (such as flight or driving simulation), 
most researchers have chosen to present a single im- 
age to both eyes, thereby halving the cost of image 
generation and permitting the use of relatively low- 
cost LCD projectors. 

The choice of whether or not to use stereo im- 
age presentation is one of many choices that influ- 
ence the fidelity of a VR device. The complexity 
of the 3-D geometric models determines the reso- 
lution of shape. Fine details are typically captured 
in textures overlaid on simple polygonal models, in 
the manner of trompe l'oeil architectural tech- 
niques of painting false facades on building sur- 
faces. The properties of these textures are important 
in determining the quality of rendered images. In 
addition, the resolution and level of anti-aliasing of 
generated images, the brightness, contrast, and res- 
olution of the display device, and even the screen 
material onto which images are projected all influ- 
ence the fidelity of the viewed images. Similar 
issues concern the fidelity of haptic displays and 
motion systems. There is little information to guide 
the architects of VR systems when deciding what 
matters most and what level of fidelity is adequate 
for their purpose. 

Notwithstanding their infidelities, VR worlds 
have been described as the "reality on your retina." 
However, VR sensory worlds differ from the real 
world at several levels, no matter how much money 
is spent on the VR setup. First, computer display 
technology is imperfect. Second, even if one could 
present sensory cues at spatial and temporal resolu- 
tion rates equal to the human visual system's, cogni- 
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tive and sensory scientists do not yet know all the 
perceptual and contextual cues needed to accurately 
recreate the experience of the real world. Because we 
do not know what all these cues are, we cannot rep- 
resent the world as accurately as we might like using 
software, hardware, and displays. 

A dozen cues to 2-D surface representation 
and 3-D structure have been identified since Fech- 
ner's initial investigations of human sensory psy- 
chophysics over 150 years ago (Palmer, 1999). 
However, others likely remain to be discovered. Sim- 
ilar problems exist for representing auditory cues, 
inertial and somatosensory cues, and olfactory cues, 
and with seamlessly synchronizing these cues. In 
many VR implementations, binocular stereopsis cues 
to 3-D structure and depth are absent, and motion 
parallax cues to structure and depth are inaccurate. 

Another signal of unreality in simulation and 
VR is the need for the viewer's eyes to accommo- 
date and converge on displays that simulate faraway 
objects that should not require ocular convergence, 
which may occur with implementations of HMDs, 
surround-screen "caves," flat-panel displays, and 
other displays that do not compensate for a closer- 
than-expected focal point. The use of collimated 
displays can help mitigate this problem by displac- 
ing the apparent location of the image into the ex- 
pected depth plane. 

At a different level, the shape and fluid motion 
of clothing and living bodies remain difficult to 
represent. The artificial surface appearances of skin 
texture, body surface geometry, and shading, and 
inaccurate deformation of body surfaces, such as 
muscle and skin, and inaccurate models of biologi- 
cal movement and facial expressions on an other- 
wise faithful reproduction of a moving human 
figure can seem disturbing and takes getting used 
to (Garau, Slater, Pertaub, & Razzaque, 2005). 

Independent of the physical appearance of a VR 
scenario represented by multisensory cues, the logic 
of the VR scenario and the behavior of autonomous 
agents such as other vehicles in a driving scenario 
may not act like real- world entities. Audiovisual fi- 
delity notwithstanding, consider how easy it is to 
detect the awkward responses of a human voice 
spoken by a computer agent designed to depict a 
human being. Current artificial beings or synthetic 
actors would likely fail Turing's test of interrogation 
for intelligence (Turing, 1950) and be distinguished 
as not human, or at least not normal. At best they 
might seem strange, mechanical, or autistic. 



Notwithstanding these shortcomings, VR can be 
used to probe the brain at work, and user reactions 
to current setups provide a window to how humans 
use a flood of sensory and cognitive cues, how these 
cues should be reproduced, how they interact, and 
what is the essence of real-world experience. 



Simulator Adaptation 
and Discomfort 

Because VR differs from reality, it can take time 
to adapt. For example, steering behavior in a real au- 
tomobile depends on sensory, perceptual, cognitive, 
and motor factors. In driving simulators, visual cues 
and steering feedback may differ from correspon- 
ding feedback in a real car (Dumas & Klee, 1996). 
For simulator experiments to be useful, drivers must 
adapt quickly to differences between the simulator 
and reality (Gopher, Weil, & Bareket, 1994; Nilsson, 
1993). Objective measurements of adaptation and 
training are needed in the VR environment. 

A study assessed time required for 80 experi- 
enced drivers to adapt to driving on simulated two- 
lane rural highways (McGehee, Lee, Rizzo, Dawson, 
& Bateman, 2004). The drivers' steering behavior 
stabilized within just 2 minutes (figure 17.5). 
Fourier analyses provided additional information on 
adaptation with respect to different frequency com- 
ponents of steering variability Older drivers' steer- 
ing behavior is more variable than younger drivers', 
but both groups adapted at similar rates. Steering is 
only one of many factors that can index adaptation 
to driving simulation, and there are many other fac- 
tors to adapt to in other types of VR applications. 

There can be a mismatch between cues associ- 
ated with the simulator or VR experience. For ex- 
ample, in a driving simulator, rich visual cues of 
self-motion (heading) without corresponding inertial 
(vestibular) cues of self-movement can create dis- 
comfort ("cybersickness" or simulator adaptation 
syndrome), analogous to the discomfort passengers 
feel in a windowless elevator or ship cabin when 
they receive strong inertial cues without correspon- 
ding visual movement cues. Related symptoms 
(Brandt & Daroff, 1979) have the potential to ad- 
versely affect a subject's performance, causing dis- 
traction, somatic preoccupation, and dropout. 

Adaptation and discomfort in VR environments 
can be studied using physiological recordings of the 
user, including electrodermal response, electrogas- 
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Figure 1 7.5. Time series of steering wheel position shows adaptation of 52 older drivers in the 
first 120 seconds of the training drive in the driving simulator SIREN. 
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trogram, and heart rate (see The Physiology of a VR 
Experience above and table 17.1). Subjective report 
is also important and can be measured with ques- 
tionnaire tools that rate the user's visual, perceptual, 
and physiological experience in the VR environment 
(Kellogg, Kennedy, & Graybiel, 1965; Kennedy, 
Lane, Berbaum, & Lilienthal, 1993). Such tools can 
be administered before and after exposure to a VR 
environment to determine how exposure increases 
discomfort relative to baseline, and how it affects 
performance and dropout. The Simulator Adapta- 
tion Questionnaire (SAQ) is a brief tool (Rizzo, 
Sheffield, Stierman, & Dawson, 2003) that incorpo- 
rates items from the Simulator Sickness Question- 
naire (SSQ; Kennedy et al., 1993) and Motion 
Sickness Questionnaire (MSQ; Kellogg et al., 1965), 
avoids redundancy, and excludes less helpful items 
(e.g., "desire to move the bowels" from the MSQ). 

One way to mitigate user discomfort is to de- 
sign VEs that minimize exposure to events that can 
trigger discomfort, such as left turns and abrupt 
braking in a driving simulator. However, this risks 
eliminating key scenarios for study. Better under- 
standing of the signs, symptoms, and factors asso- 
ciated with adaptation to VR environments is 
needed, including the role of presence, immersion, 
visual aftereffects, vestibular adaptation, display 
configuration, and movement cues. The efficacy of 



scopolamine patches, mild electrical stimulation of 
the median nerve "to make stomach rhythms re- 
turn to normal," and the extent to which move- 
ment cues can reduce discomfort are under 
investigation (Mollenhauer, 2004). 

Greenberg (2004) pointed out that the goal of 
characterizing a simulator's fidelity is to identify the 
major artifacts and how they matter to a given VR 
experiment. A VR study should be undertaken 
only after the investigator understands the artifacts 
and can control or adjust adequately for them. 
Most neuroergonomics research questions cannot 
be addressed using VR unless there is a useful level 
of accuracy. In certain settings, the accuracy need 
not be high and a useful simulation may even be 
surrealistic (see below). 

Understanding the level of accuracy of VR de- 
pends on knowledge of the most important charac- 
teristics of physical fidelity in the VR cues, including 
visual, auditory, somatosensory, vestibular (inertial), 
and even olfactory cuing systems. This depends on a 
clear understanding of the system being replicated 
(e.g., driving, flying, phobias) and on the accuracy 
of reproduction of appliances in the laboratory set 
up to interact with the user, such as the cab in a 
driving simulator or the cockpit in a flight simulator 
or the HMD or glove. Another factor affecting the fi- 
delity (and acceptance) of the VR setup is the intra- 
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siveness of devices used to present the stimuli and 
measure user responses, including physiological 
(such as EEG, eye movement recording systems, and 
galvanic skin response; see below). Greenberg 
(2004) summarized relevant factors for the assess- 
ment of the fidelity of driving simulation along 21 
dimensions in three main domains: the visual world, 
terrain and roadways, and vehicle models. 



How Low Can You Go? 

In VR or simulation, a real-life task is reduced to a 
lower-dimension approximation. One theory is 
that transfer between the simulated and real-life 
tasks will occur to the extent that they share com- 
mon components. More relevant, perhaps, is the 
level of psychological fidelity or functional equiva- 
lence of the simulation (Alessi, 1988; Lintern, 
1991; Lintern, Roscoe, Koonce, & Segal, 1990). 
Replicating key portions of a task convincingly 
and with enough fidelity to immerse, engage, in- 
terest, and provide presence may matter more 
than reproducing exactly what is out there in the 
real world. 

A common approach to VR strives for 
computer-generated photorealistic representations 
(Brooks, 1999) using multiple large display screens 
or HMDs yielding 150- to 360-degree fields of view 
and providing optical flow and peripheral vision 
cues not easily achieved on a single small display. Yet 
the high cost and technical complexity of operating 
and maintaining these systems, including software 
provenance, legacy, and updates, limits their use to 
large university, government, or corporate research 
settings, and they cannot be practically deployed in 
physicians' or psychologists' offices for clinical ap- 
plications (e.g., the use of VR to assess drivers who 
are at risk for crashes due to cognitive impairments). 
Moreover, scenario design in VR settings has been 
ad hoc and unfocused. 

Early driving simulators created video game- 
like scenarios, and some operators felt discomfort, 
possibly because low microprocessor speeds in- 
troduced coupling delays between visual motion 
and driver performance (Frank, 1988). Yet, despite 
modest degrees of realism, such simulations suc- 
cessfully showed how operators in the loop, such 
as pilots and drivers running hand and foot con- 
trols, are affected by secondary task loads, fatigue, 
alcohol intoxication, aging, and cognitive impair- 
ments (Brouwer, Ponds, Van Wolffelaar, & Van- 



Zomeren, 1989; de Waard & Rooijers, 1994; Din- 
gus, Hardee, & Wierwille, 1987; Guerrier, Mani- 
vannan, Pacheco, & Wilkie, 1995; Haraldsson, 
Carenfelt, Diderichsen, Nygren, & Tingvall, 1990; 
Katz et al., 1990; McMillen & Wells-Parker, 1987; 
Rizzo, McGehee, Dawson, & Anderson, 2001). 
Even environments with very impoverished sen- 
sory cues can provide a useful basis for studying 
perception and action. For example, a study of the 
perception of direction of heading examined the 
role of motion cues in driving (Ni, Andersen, 
McEvoy, & Rizzo, 2005). A variety of cues provided 
crucial inputs for moving through the environment. 
Optical flow is the apparent motion of object points 
across the visual field as an observer moves through 
an environment. It is a well-studied cue for the per- 
ception and control of locomotion (Gibson, 1966, 
1979). The optical flow field can be conceptualized 
as an image of vectors that represent how points are 
moved from image to image in an image sequence. 
Visual landmarks also provide key information for 
the perception and control of locomotion (Hahn, 
Andersen, & Saidpour, 2003) and are important for 
movement perception in real environments (see 
chapter 9) and virtual environments. 

Ni et al. (2005) used a tracking task in which 
younger and older drivers viewed optical flow 
fields comprising achromatic dots that were ran- 
domly positioned in 3-D space, as with a star field 
(see figure 17.6). The task simulated motion along 
the subject's line of sight displaced laterally by a 
forcing function (resembling unpredictable vehicle 
sidewind on a windy road). The driver had to com- 
pensate for the sidewind using a steering wheel so 
that the path of locomotion was always straight. 
On half of the trials, landmark information was 
given by a few colored dots within the flow field of 
achromatic dots. Steering control was indexed by 
tracking error (root mean square) and coherency 
(the squared correlation between the input and re- 
sponse at a particular frequency). Independent 
measures were number of dots in the optical flow 
field, landmark information (present or not), and 
frequency of lateral perturbation. Results suggested 
greater accuracy and less steering control error for 
younger subjects (Ni et al., 2005). Both groups 
showed improvement with an increase in optical 
flow information (i.e., an increase in the density of 
dots in the flow field). Younger subjects were more 
efficient at using optical flow information and 
could use landmark information to improve steer- 
ing control. This study illustrates how a perceptually 
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Figure 1 7.6. Heading from optical flow. 
In this schematic illustration, each vector 
represents the change in position of a 
texture element in the display during for- 
ward motion. The subject's task was to 
judge the direction of observer motion 
relative to the vertical bar. In the illustra- 
tion, the correct response is the direction 
to the right of the bar. 




impoverished environment can replicate key ele- 
ments involved in real-world navigation. 

Although VR environments differ from natural 
environments, not all differences matter. Different 
physical stimuli known as metamers produce indis- 
tinguishable neural or psychological states (Brindley, 
1970). Humans "complete" images across the nor- 
mal physiological blind spot (caused by the lack of 
retinal receptors where the optic nerve meets the 
back of the eye). They may fill in gaps and missing 
details from the area of an acquired visual field defect 
and fail to recognize that they have a deficit (Rizzo & 
Barton, 1998). Normal observers may seem to be 
blind to glaring details that defy everyday expecta- 
tions, such as the famous change-blindness demon- 
stration showing that observers with normal vision 
failed to notice a gorilla walking through a group of 
people tossing a ball (Simons & Chabris, 1999). 
Consequently, they may not be disturbed by some 
missing details or low fidelity in VR environments. 

Low fidelity can be a problem if it reduces im- 
mersion in a simulated task. One approach to in- 
crease immersion is to place real objects in VEs as 
props. The touch of a physically real surface or 
simply feeling a ledge can do much to heighten the 
sense of immersion (Insko, 2001; Lok, Naik, Whit- 
ton, & Brooks, 2003). The use of simple objects as 
props to augment high-fidelity visual representa- 
tions compellingly demonstrates the power that 
small cues have to give participants a sense of real- 
ness and presence. For example, a wooden plank 
can be used to create an apparent ledge at the edge 
of a virtual precipice. 



A key question is, is it fair to cheat (i.e., reduce, 
enhance, or augment some cues) to increase users' 
immersion and decrease discomfort? For example, is 
it acceptable to substantially enlarge road signs in a 
driving simulator to compensate for poor acuity 
caused by low-resolution displays? The fidelity- 
related question of how closely a simulated world 
should match the real world depends on the goals of 
the simulation and requires multilevel considerations 
of and comparisons between simulated cues (e.g., vi- 
sual, auditory, haptic, and movement cues) and tasks 
and corresponding real-world cues and tasks. Poten- 
tial reasons for exploring the lower boundaries of 
fidelity besides economy are that increasing levels of 
fidelity may limit data collection, dilute training 
effects, undermine experimental generalizations, and 
increase the likelihood of simulator discomfort. 



Nonrealistic Virtual Environments 

VEs need not be highly realistic to be effective for 
experimental and training applications in neuroer- 
gonomics. A number of researchers have examined 
the use of deliberately unreal VR environments as 
surrogates for real environments. This line of re- 
search parallels developments in nonphotorealistic 
rendering (NPR) techniques for static and ani- 
mated graphics. NPRs abstract the essential con- 
tent in an image or image stream and often can be 
produced very efficiently. Simple drawings are of- 
ten more informative than photographs. For exam- 
ple, instruction manuals frequently make use of 
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schematics to illustrate assembly procedures, and 
medical textbooks are filled with hand-drawn illus- 
trations of anatomical structures (Finkelstein & 
Markosian, 2003). 

Fischer, Bartz, and Strasser (2005) examined 
the use of NPR to improve the blending of real and 
synthetic objects in AR by applying a painterly 
filter to produce stylized images from a video cam- 
era. The filtered video stream was combined with 
computer-generated objects rendered with a non- 
photorealistic method. The goal was to blur the dis- 
tinction between real and virtual objects. The 
resultant images are similar to hand-drawn sketches 
(Fischer et al., 2005). Sketches and line drawings 
can convey a sense of volume and depth while re- 
ducing distracting imperfections and artifacts of 
more realistic renderings. Gooch and Willemsen 
(2002) examined distance judgments for VR envi- 
ronments rendered with line drawings on an HMD. 
Subjects were asked to view a target on the ground 
located 2-5 meters away, close their eyes, and walk 
to the target. They undershot distances relative to 
the same walking task in a real environment, which 
corresponded with undershoot of similar experi- 
ments conducted with more realistically rendered 
VR environments in HMDs. 

Severson and colleagues (Rizzo, Severson, Cre- 
mer, & Price, 2003; Rizzo, 2004) developed an ab- 
stract VR environment tool that captures essential 
elements of driving-related decision making and is 
deployable on single-screen PC systems (figure 
17.7). This tool tests go/no-go decision making. It is 
based on a surreal environment and provides perfor- 
mance measures along the lines of a neuropsycho- 
logical test. The nonconventional design approach 
was motivated by results of synthetic vision display 
system research for navigation and situational aware- 
ness. Designers focused on what was needed for the 
intended assessment rather than assuming visual re- 
alism was necessary and used abstract, rather than 
photorealistic, representations in the VE (Ellis, 1993, 
2000). This approach draws from perceptual psy- 
chology, computer graphics, art, and human factors, 
and provides sufficient pictorial and motion cues rel- 
evant to the task of perceiving the spatial relation- 
ships of objects and the orientation of the user in the 
VE (Cutting, 1997; Palmer, 1999; Wanger, Ferwerda, 
& Greenberg., 1992). Similar to high-fidelity driving 
simulators and avionics synthetic vision systems, the 
go/no-go tool provided motion parallax, optical flow, 
shading, texture gradients, relative size, perspective, 



occlusion, convergence of parallel lines, and posi- 
tion of objects relative to the horizon. Application 
of the go/no-go tool showed significant differences 
in errors between 16 subjects with neurological 
impairments (14 with focal brain lesions, 2 with 
Alzheimer's disease) and 16 neurologically normal 
subjects. The finding of a shallower learning curve 
across go/no-go trials in brain-damaged drivers 
suggests a failure of response selection criteria based 
on prior experience, as previously reported in 
brain-damaged individuals with decision-making 
impairments on the Gambling Task (e.g., Bechara, 
Damasio, Tranel, & Damasio, 1997; see above). 

These results suggest that VR can be used to 
develop new neuropsychological tests. VR tasks 
can provide a window on the brain at work in 
strategic and tactical situations encountered in 
the real world that are not tested by typical neu- 
ropsychological tests. A PC-based VR tool can dis- 
tinguish decision-making-impaired people, which 
traditional neuropsychological test batteries may 
not (Damasio & Anderson, 2003). 



Validation and Cross-Platform 
Comparisons 

The results of experiments conducted in VR are 
valid only to the extent that people behave the same 
in the real world as they do in the virtual world. 
Likewise, VR training will be effective only if learned 
behaviors transfer to the real world. The most direct 
way to test the validity of a study conducted in a VR 
environment is to compare the results to perfor- 
mance under the same conditions in the real world. 
However, in many cases it is not possible to conduct 
comparable tests in the real world because to do so 
would expose subjects to dangerous circumstances, 
posing a risk of injury. Even if a methodology is vali- 
dated for a particular VE, the question that remains 
is how that methodology transfers to a different VE. 
VE installations vary in a multitude of ways, includ- 
ing the display technology, the rendering software, 
the interaction devices, and the content presented 
(i.e., object models and scenarios). In order to be as- 
sured that results generalize to a variety of platforms, 
it is important to conduct cross-platform compara- 
tive studies to determine what differences influence 
behavior. 

A recent body of research has examined whether 
or not fundamental aspects of visual perception and 
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Figure 1 7.7. (Top) The go/no-go tool used a nonphotorealistic representation of a 3-D 
virtual space. Visual and pictorial cues provided situational awareness in a small field 
of view, similar to the design constraints placed on aviation research display system re- 
searchers (Theunissen, 1997). (Bottom) At point T, gate-closing trigger point A (easy), 
B (medium), or C (difficult) is computed, based on current speed, deceleration limit, 
and other parameters. Each subject drove through a series of intersections (marked by 
Xs), which had gates that opened and closed. When the driver reached point T, 100 
meteres before an intersection, a green Go or red Stop signal appeared at the bottom of 
the display and a gate-closing trigger point (A, B, or C) was computed based on a de- 
celeration constant, gate closure animation parameters, driver speed, and amount of 
time allotted to the driver to make a decision. 



action are preserved in VR environments. For exam- 
ple, do people judge distances and motions the same 
in VEs as they do in real environments? A number 
of studies have found that people underestimate 
distances in VR by as much as 50% relative to com- 
parable judgments in real environments (Loomis & 
Knapp, 2003; Thompson et al., 2004; Willemsen 
& Gooch, 2002). For example, Loomis and Knapp 
compared distance estimates in the real world to dis- 
tance estimates in VR presented on an HMD. Sub- 
jects viewed a target on the ground. They then closed 
their eyes, turned 90 degrees, took a small number 



of steps, and then pointed to where they thought the 
target would be. On average, subjects underesti- 
mated the true angle by a factor of 2 in VR relative to 
the same task in a corresponding real environment. 

Contradicting these results, a number of studies 
have found distance estimates to be similar in real 
and virtual environments (Interrante, Anderson, & 
Ries, 2004; Plumert, Kearney, & Cremer, 2005; 
Witmer & Sadowski, 1998). Plumert et al. (2005) 
examined distance estimation in large-screen virtual 
environments. Subjects judged the distance of a 
person standing on a grassy lawn in a real environ- 
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ment and an image of a person placed on a textured 
model of the outdoor environment presented on 
a panoramic display. Perceived distance was esti- 
mated by the time it took subjects to perform an 
imagined walk to the target. Subjects started a stop- 
watch as they initiated their imagined walk and 
stopped it when (in their imagination) they arrived 
at the target. They found that time-to- walk esti- 
mates were very similar in real and virtual environ- 
ments. In general, subjects were accurate in judging 
the distance to targets at 20 feet and underesti- 
mated the distance to targets at 40 to 120 feet. 

If distance perception is significantly distorted 
in VR, then performance on tasks that require 
judgment of distance is likely to be very different in 
VR than in the real world. This could profoundly 
influence the outcomes of experiments in VR. 
However, the conditions under which distance is 
underestimated in VR and the cause of this distor- 
tion remain unclear and are under investigation. 

The validity of a simulation can be assessed by 
comparing results in VR to a variety of gold stan- 
dard real-world outcomes. For example, the behav- 
iors of drivers in a ground vehicle simulator could 
be compared to state records of crashes and moving 
violations of real people driving real cars on real 
roads. This epidemiological record can be assessed 
prospectively, but there are few direct observations, 
and records are reconstructed from imperfect and 
incomplete accounts by untrained observers in 
chaotic conditions. It is now also possible to mea- 
sure real behavior in the real world (see chapter 8). 
Results make it clear that the epidemiological record 
may not accurately reflect the truth. In one remark- 
able example, an innocent driver was charged with 
an at-fault, rear-end collision after his car was struck 
by an oncoming car that spun 180 degrees and 
veered into his lane (Dingus et al., 2005). 

Can a standard VR scenario that is run using dif- 
ferent hardware and software be expected to give 
similar results in similar populations of subjects? 
This question is relevant to developing standard VR 
scenarios and performing clinical trials across multi- 
ple research sites. In this vein, the HASTE group 
considered several specific criteria for cross-platform 
comparisons in VR scenarios implemented in a 
range of driving simulators. Key comparison mea- 
sures included minimum speed and speed variation, 
lateral position and lateral position variation, lane 
exceedances, time to line crossing, steering wheel re- 
versal rate, time to collision, time headway, distance 
headway, and brake reaction time. Optional com- 



parison measures were high-frequency component 
of steering wheel angle variation, steering entropy, 
high-risk overtakings, abrupt onsets of brakes, and 
postencroachment time (HASTE, 2005). 



Guidelines, Standards, 
and Clinical Trials 

A current problem in using VR in neuroergonomics 
research is lack of guidelines and standards. Guide- 
lines are recommendations of good practice that 
rely on their authors' credibility and reputation. 
Standards are formal documents published by 
standards-making bodies that develop through ex- 
tensive consultation, consensus, and formal voting. 
Guidelines and standards help establish best prac- 
tices. They provide a common frame of reference 
across VR (simulator and scenario) designs over 
time to avoid misunderstanding and to allow dif- 
ferent user groups to connect. They allow review- 
ers to assess research quality, to compare results 
across studies for quality and consistency, and to 
help identify inconsistencies and the need for fur- 
ther guidelines and standards. 

The lack of guidelines and standards is hinder- 
ing progress in human factors and neuroergonom- 
ics research and has emerged as a major issue in VR 
setups for driving simulation. VR can be an impor- 
tant means for safe and objective assessment of 
performance capabilities in normal and impaired 
automobile drivers. Yet it remains a cottage indus- 
try of homegrown devices with few guidelines and 
standards. 

Having guidelines and standards for simulation 
and VR scenarios can facilitate comparisons of oper- 
ator performance across different VR platforms, in- 
cluding the collection of large amounts of data at 
different institutions with greater power for address- 
ing worldwide public health issues conducive to re- 
search using VR. Comparisons against standards can 
clearly reveal missing descriptors and other weak- 
nesses in ever-mounting numbers of research reports 
on driving simulation and other applications of VR. 

In driving simulation, the scenario might focus 
on key situations that convey a potential high crash 
risk, such as intersection incursion avoidance and 
rear-end collision avoidance scenarios involving 
older drivers and scenarios that address behavioral 
effects of using in-vehicle devices such as cell 
phones and information displays. In a training appli- 
cation, the scenario might focus on risk acceptance 
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Table 17.2. Sample Driving Scenario Description 



Script 



Cognitive constructs stressed 
Dependent measures 



Independent variables 



Implementation challenges 



Measurement challenges 



Testing validity of scenario 



The driver is traveling on a road with two lanes of traffic, each moving at 

different speeds. At different times, one lane of traffic is moving more 

advantageously (faster), although overall this may be the slower lane. The 

drivers task is to pass through traffic as quickly as possible. 

This task would be similar to the gambling task (Bechara et al, 1997) in which 

an individual has to overcome an immediate reward to ensure long-term 

benefit. Variations can be made on this script: the two lanes of traffic could 

average the same speed or could even be moving at the same speed. The 

drivers perception that one lane is moving faster may be a visual and cognitive 

illusion. 

Attention, perception, and decision making. 

Time it takes the driver to maneuver through the traffic to arrive at a 

destination, number of navigation errors, and number of moving violations or 

safety errors (e.g., excessive speed, near misses, and collisions) could be 

recorded. 

The number of vehicles involved, number of lanes, speed of the different lanes 

and final destination, and travel contingencies could be manipulated (e.g., the 

driver could be instructed to get off at a specific exit for a hospital). 

It may be difficult to create a realistic feel. The problem may be lessened by 

giving the driver external instructions, thus altering driver expectations and 

rewards or incentives. For instance, the instructions could be to drive as if you 

were taking a pregnant woman or a critically sick person to the emergency 

room. Is it possible for the driver to be able to change lanes when desired? 

How will the surrounding cars respond to the driver? 

Some drivers will not try to get into the faster lane, if they think it makes little 

difference in the long run. A questionnaire following the task may be helpful 

in assessing how fast the driver felt each lane was moving and whether or not 

the driver would have changed lanes if given the opportunity. Variations in 

personalities would have to be considered. 

An instrumented vehicle could be used to study lane-change behavior during 

times of high-density traffic, yet the environmental variables could not be 

easily controlled. 



in teenage drivers. In a medical application, the 
scenario might focus on a high-risk situation in 
anesthesia. In a psychological application, the sce- 
nario might be tailored to desensitize a patient to a 
certain phobia (see below). 

Task analysis (Hackos & Redish, 1998) can de- 
construct and help clarify the logical structure of 
complex VR scenarios, the detailed flow of actions 
a subject has to take in a VR task, and related issues 
such as operational definitions and measurements 
of variables. This can facilitate cross-platform com- 
parisons, help create a body of shared VR scenarios 
for multicenter research trials, and contribute to an 
infrastructure for worldwide research on a variety 
of global public health issues. 

A proposed format for describing a VR scenario 
for neuroergonomics research in traffic safety is 
shown in table 17.2. An example specification is 
shown for a lane-change scenario that includes a 
script, the cognitive constructs stressed, dependent 



measures, independent variables, implementation 
challenges, plan for testing the validity of the sce- 
nario, and a bibliography of related literature. 
Similar strategies could be considered for other VR 
applications. 

Key topics for guidelines and standards in VR 
are (1) scenario design; (2) selection and definition 
of performance measures; (3) standards for report- 
ing experimental setup and results; (4) subject selec- 
tion criteria; (5) reporting of independent measures; 
(6) standards for physiological recording; (7) graph- 
ics, audio, and movement; (8) training criteria; 
(9) subject adaptation and comfort; and (10) crite- 
ria for validation. 

Making guidelines and standards involves sci- 
ence, logic, policy, and politics. Standards may 
seem to favor one group over another. Key persons 
may seem to exert undue influence over the final 
product. The structure and formality of standards 
might seem to hinder innovation. It may be easier 
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to propose standards than to apply them in prac- 
tice. For example, how well can most driving 
simulator scenarios be described within a common 
framework? Should researchers be expected to im- 
plement a key VR scenario just one way? Can levels 
of fidelity be adequately specified for cross-platform 
comparisons? Can we overcome vested interests 
and entrenched opinions? These are questions that 
demand attention and a call for greater action. 

Having VR standards would facilitate multicen- 
ter clinical trials in a variety of settings in which VR 
has shown promise, such as driving simulation, 
treatment of phobias, and so on. A clinical trial has 
"some formal structure of an experiment, particu- 
larly control over treatment assignment by the in- 
vestigator" (Piantadosi, 1997). According to the 
National Institutes of Health (2005), "A clinical 
trial ... is a research study in human volunteers to 
answer specific health questions. [Clinical trials] are 
the fastest and safest way to find treatments that . . . 
improve health. Interventional trials determine 
whether experimental treatments ... are safe and 
effective under controlled environments." 

For instance, driving simulator scenarios are 
becoming key elements in state and federal efforts 
to probe (1) the efficacy of novice and older driver 
training programs; (2) fitness to drive in patients 
with performance declines due to aging and mild 
cognitive impairment, traumatic brain injury, neu- 
rodegenerative disorders such as Alzheimer's and 
Parkinson's diseases, and sleep disturbances; and 
(3) acute and chronic effects of many medications 
such as analgesics, antidepressants, psychostimu- 
lants, antidepressants, and cancer chemotherapy 
agents. To make meaningful comparisons between 
research studies conducted on differing platforms, 
it is essential that the scenarios used be specified in 
sufficient detail to implement them on simulators 
that depend on very different hardware, visual 
database and scenario development tools, image 
rendering software, and data collection systems. 



Future Directions 

VR has many potential applications in neuroer- 
gonomic assessments of healthy and impaired op- 
erators. VR has been used to assess the effects of 
introducing new technology to the human operator 
in the loop: cell phones, GPS, infotainment, auto- 
mated highways, heads-up displays, and aircraft 
displays. Human performance measurements in 



VEs include motor learning and rehabilitation, 
and information visualization in VR (glass cockpit, 
information-gathering behaviors of military or 
homeland security analysts, and such). VR can be 
used in applications to train physicians effectively, 
resulting in fewer medical errors. To assess the effi- 
cacy of applications that use VEs as training simu- 
lators, we must measure transfer of training effects. 

VR applications are being proposed more and 
more as a tool for assessing and treating a variety of 
neurosychological, psychiatric, and physical disor- 
ders. These applications pertain to planning and de- 
cision making, spatial orientation and navigation, 
mood disorders, anxiety, panic disorder, agorapho- 
bia, and eating disorders. The feeling of being some- 
where else in VR may help people suffering under 
drug addiction, dental extractions, chemotherapy, 
dialysis, physical therapy, or other discomforts. 

There are numerous research groups providing 
insights and therapeutic use of VR as a tool for 
therapy and rehabilitation of panic disorders and 
post-traumatic stress disorders, and as an analgesic 
during therapeutic and rehabilitation treatment 
(Hodges et al., 2001; Hoffman Garcia-Palacios, 
Carlin, & Botella-Arbona, 2003; Hoffman, Richards, 
Coda, Richards, & Sharar, 2003; Hoffman et al., 
2004; Rothbaum & Schwartz, 2002; Rothbaum 
et al, 1995; Wiederhold & Wiederhold, 2004). VR 
therapy is an ongoing area of exploration, currently 
with limited clinical use, but has been embraced by 
a small community of providers. Beyond the chal- 
lenges of utilizing high-tech computing technology 
in a therapy setting, there are operational chal- 
lenges in the use of VR therapy: with customization 
of environments for the broad range of scenarios 
required to meet a subject's therapeutic needs; and 
with the limited educational resources for the de- 
sign and development of scenarios, environments, 
and protocols and the training of therapists to uti- 
lize the techniques. For example, in traditional 
therapy sessions, subjects would utilize in vivo ex- 
posure or be required to imagine a scenario and set- 
ting in which their phobia would take place 
(Hodges, Rothbaum, Watson, Kessler, & Opdyke, 
1996). Utilizing VR therapy, an appropriate sce- 
nario and environment would need to be available 
to the therapist and patient for treatment of the 
disorder. For each scenario and environment, the 
efficacy and usability of the VR scenario would 
need to be validated for clinical use. As a therapeu- 
tic modality, VR therapy appears to manifest a num- 
ber of the challenges noted throughout this chapter. 
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Clinical application challenges notwithstand- 
ing, there are particularly compelling examples of 
research into the use of VR therapy. The University 
of Washington HITLab is exploring VR therapy as a 
nonpharmacological analgesic for patients receiv- 
ing burn treatment therapy, and it is conducting 
exploratory research utilizing synchronized VR ex- 
posure with fMRI brain scans to study the neural 
correlates of psychological disorders and the im- 
pact of therapy on patterns of fear-related brain 
activity (Hoffman et al., 2004). 

Patients with anxiety disorders, post-traumatic 
stress disorders or phobias (such agoraphobia (fear 
of crowds), arachnophobia (fear of spiders), acro- 
phobia (fear of heights), and aviophobia (fear of 
flying) might be desensitized by exposure to virtual 
threats. Burn victims may be distracted from their 
pain by moving through icy VR worlds with snow- 
men, igloos, and penguins in association with 
modulation of brain activity (Hoffman et al., 2004). 
Patients with eating disorders might be treated by 
exposure to differing body images or renditions of 
virtual dining experiences. 

Advances in the technology such as HMDs, in- 
teractive gloves, and the Internet will facilitate pene- 
tration of VR in these applications. Wireless 
movement trackers (see chapter 8) will lead to new 
and better interfaces for users to manipulate objects 
in VR. Improvements in cost and fidelity may be 
driven by the resources of the gaming industry. VR 
users may be connected to distributed virtual worlds 
for collaborative work and play over the Internet. 

Honeywell's use of a gaming platform for as- 
sessing the performance of research tools and user 
performance provided a visually rich VE and com- 
plex scenario capabilities that would be challenging 
for most researchers to create from scratch (Dor- 
neich et al., 2004). The marriage of VR and gaming 
has been a growing collaboration fostered by mili- 
tary funding of computer-based simulation and 
training and the popularity and resources for devel- 
opment of first-person-shooter and distributed 
gaming (Zyda & Sheehan, 1997). 

One study suggested that action video game 
playing enhances visual attention skills in habitual 
video game players versus non-video game players 
(Green & Bavalier, 2003). Nonplayers trained on 
an action video game showed improvement from 
their pretraining abilities, establishing the role of 
playing in this effect. Further work is needed to in- 
vestigate this effect and address whether immersion 



in VR through video game playing can enhance 
other aspects of sensation, memory, and other cog- 
nitive processes. 

Avatars and simulated humans will assume 
more realistic traits, including movement, speech, 
and language, and will seem to understand and re- 
spond appropriately to what we say and do. This 
may result in virtual friends, advisors, and thera- 
pists (e.g., on long space flights, in lonely outposts). 



Conclusion 

Multidisciplinary vision and collaboration has es- 
tablished VR as a powerful tool to advance neu- 
roergonomics. Researchers are creatively exploring 
and expanding VR applications using systems that 
vary from high-fidelity, motion-based simulation 
and fully immersive, head-mounted VR to PC- 
based VR gaming platforms. These systems are pro- 
viding computer-based, synthetic environments to 
train, treat, and augment human behavior and cog- 
nitive performance in simulacra of the real world 
and to investigate the relationships between per- 
ception, action, and consciousness. 

Challenges remain. Although the term VR gen- 
erally describes computer-based environments, 
there are no clearly defined standards for what the 
systems should provide in terms of fidelity, display 
technology, or interaction devices. Standards are 
needed for assessing research outcomes and repli- 
cating experiments in multi-institution research. 
The current vacuum of standards can be at least 
partly attributed to the rapid pace of change. VR is 
an evolving, fluid, and progressing medium that 
benefits from the advances in PC-based computing 
and graphics-rendering capabilities. 



Main Points 

1 . VR applications show promise for studying the 
behavior of people at work, for training 
workers, and for improving worker perfor- 
mance through augmented reality and 
augmented cognition environments. 

2. The results of simulator scenarios in different 
labs with different VR environments should be 
compared with each other, as well as with the 
results in comparable tasks in real-world 
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settings, in order to test the replicability and 
validity of the results. 
3. VR environments can be useful even if they 
depart significantly from reality. Even surreal 
VR environments can provide useful 
information on the brain and behavior at 
work. 
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Cynthia Breazeal and Rosalind Picard 



The Role of Emotion-Inspired 
Abilities in Relational Robots 



This chapter presents our motivation and a snap- 
shot of our work to date in building robots with 
social-emotional skills, inspired by findings in neu- 
roscience, cognitive science, and human behavior. 
Our primary motivation for building robots with 
social-emotional-inspired capabilities is to develop 
"relational robots" and their associated applications 
in diverse areas such as health, education, or work 
productivity where the human user derives perfor- 
mance benefit from establishing a kind of social 
rapport with the robot. We describe some of the fu- 
ture applications for such robots, provide a brief 
summary of the current capabilities of state-of-the- 
art socially interactive robots, present recent find- 
ings in human-computer interaction, and conclude 
with a few challenges that we would like to see ad- 
dressed in future research. Much of what we de- 
scribe in this chapter cannot be done by most 
machines today, but we are working to bring about 
research breakthroughs that will make such things 
possible. 

To many people, the idea of an emotional robot 
is nonsensical if not impossible. How could a ma- 
chine ever have emotions? And why should it? After 
all, robots today are largely automated machines 
that people use to perform useful work in various 
settings (see also chapter 16, this volume). Manufac- 



turing robots, for instance, tirelessly perform simple 
repetitive tasks. They are large and potentially dan- 
gerous, and therefore are located safely apart from 
people. Autonomous and semiautonomous mobile 
robots have had successes as probes for exploring 
remote and hazardous environments, such as plane- 
tary rovers, oil well inspection robots, or urban 
search and rescue robots. These successes have all 
come without explicitly building emotion into any 
of these machines. One might even argue that the 
fact that these machines lack emotion is a benefit. 
For example, a state like boredom could interfere 
with the ability to perform the simple repetitive 
tasks that are, for many machines, their raison d'e- 
tre. And who really needs an emotional toaster? 

The above views, while quite reasonable, reveal 
only part of the story. A new understanding of 
emotion from neuroscience discoveries has led to 
significant rethinking of the role of emotion in in- 
telligent processes. While the popular view of emo- 
tion is still generally associated with undesirable 
episodes of "being emotional," implying a state of 
imbalance and reduced rationality in thinking, the 
new scientific view, rooted in growing evidence, 
is a more balanced one: Emotion facilitates flexi- 
ble, rational, useful processing, especially when an 
entity faces complex unpredictable inputs and has 
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to respond with limited resources in an intelligent 
way (e.g., Bechara, Damasio, Tranel, & Damasio, 
1997; Damasio, 1994; Le Doux, 1996; Panksepp, 
1998; see also chapter 12, this volume). Not only 
is too much emotion detrimental, but so too is too 
little emotion. Emotion mechanisms are now seen 
as regulatory, biasing cognition, perception, deci- 
sion making, memory, and action in useful ways 
that are adaptive and lead to intelligent real-time 
behavior. Mechanisms of emotion, when they are 
functioning appropriately, do not make people ap- 
pear emotional in the negative sense of the word. 



interrupt cycles, which essentially hijack a ma- 
chine's routine processing, function analogously to 
the fear system in an animal). Moreover, many of 
the problems today's unintelligent machines have 
difficulty with bear some similarity to problems 
faced by people with impaired emotional systems 
(Picard, 1997). There is a growing number of rea- 
sons why scientists might wish to consider emotion- 
like mechanisms in the design of future intelligent 
machines. 



Social Interaction with Humans 



Intelligence in Uncertain 
Environments 

With this new understanding of the valuable role 
of emotion in people and animals comes a chance 
to reexamine the utility it might provide in ma- 
chines. If emotion is as integral to intelligent real- 
time functioning as the evidence suggests, then 
more intelligent machines are likely to need more 
sophisticated emotion-like mechanisms, even if we 
never want the machines to appear emotional. 
Much like an animal, an autonomous mobile robot 
must apply its limited resources to address multi- 
ple concerns (performing tasks, self-preservation, 
etc.) while faced with complex, unpredictable, and 
often dangerous situations. For instance, balancing 
emotion-inspired mechanisms associated with in- 
terest and fear could produce a focused yet safe 
searching behavior for a routine surveillance robot. 
For this application, one could take inspiration 
from the classic example of Lorenz regarding the 
exploratory behavior of a raven when investigating 
an object on the ground starting from a perch high 
up in a tree. For the robot, just as for the raven, in- 
terest encourages exploration and sustains focus on 
the target, while recurring low levels of fear moti- 
vate it to retreat to safe distances, thereby keeping 
its exploration within safe bounds. Thus, an analo- 
gous exploratory pattern for a surveillance robot 
would consist of several iterative passes toward the 
target: On each pass, move closer to investigate the 
object in question and return to a launching point 
that is successively closer to the target. 

In fact, mechanisms of emotion, in the form 
of regulatory signals and biasing functions, are 
already present in some low-level ways in many 
machine architectures today (e.g., emergency 



There is another domain where a need for social- 
emotional skills in machines is becoming of im- 
mediate significance, and where machines will 
probably need not only internal emotion-like regu- 
latory mechanisms, but also the ability to sense 
and respond appropriately to your emotions. We 
are beginning to witness a new breed of autonomous 
robots called personal service robots. Rather than op- 
erating far from people in hazardous environments, 
these robots function in the human environment. 
In particular, consumer robots (e.g., robot toys and 
autonomous vacuum cleaners) are already success- 
ful products. Reports by UNEC and IFR (2002) 
predict that the personal robot market — robots 
designed to assist, protect, educate, or entertain in 
the home — is on the verge of dramatic growth. In 
particular, it is anticipated that the needs of a grow- 
ing aging population will generate a substantial de- 
mand for personal robots that can assist them in 
the home. Such a robot should be persuasive in 
ways that are sensitive to people, such as remind- 
ing them when to take medication, without being 
annoying or upsetting. It should understand what 
the person's changing needs are and the urgency 
for satisfying them so that it can set appropriate 
priorities. It needs to understand when the person 
is distressed or in trouble so that it can get help if 
needed. Furthermore, people should enjoy having 
the robot in their life because it is useful and pleas- 
ant to have around. The issue of enduring con- 
sumer appeal, long after the novelty wears off, is 
important not only for the success of robotic prod- 
ucts, but also for the well-being of the people for 
whom they will serve as companions. 

Yet the majority of robots today treat people ei- 
ther as other objects in the environment to be navi- 
gated around or at best in a manner characteristic 
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of socially impaired people. Robots do not under- 
stand or interact with you any differently than if 
you were a television. They are not aware of your 
goals and intentions. As a result, they do not know 
how to appropriately adjust their behavior to help 
you as your goals and needs change. They do not 
flexibly draw their attention to what you currently 
find of interest in order to coordinate their behav- 
ior with yours. They do not realize that perceiving 
a given situation from different visual perspectives 
impacts what you know and believe to be true 
about it. Consequently, they do not bring impor- 
tant information to your attention that is not easily 
accessible to you when you need it. They are not 
aware of your emotions, feelings, or attitudes. As a 
result, they cannot prioritize what is the most im- 
portant to do for you according to what pleases 
you or to what you find to be most urgent, relevant, 
or significant. Although there are initial strides in 
giving robots these social-emotive abilities (see 
Breazeal, 2002; Fong, Nourbakshsh, & Dauten- 
hahn, 2002; Picard, 1997), there remains quite a 
lot of work to be done before robots are socially 
and emotionally intelligent entities. 

Our interaction with technology evokes strong 
emotional states within us, unfortunately often 
negative. Many people have experienced the frus- 
tration of a computer program crashing in the mid- 
dle of work, trying to customize our electronic 
gadgets through confusing menus, or waiting 
through countless phone options as we try to book 
an airline flight. Recent human-computer interac- 
tion studies have found that the ways in which our 
interaction with technology influences our cogni- 
tive and affective states can either help or hinder 
our performance. For instance, Nass et al. (2005) 
found that during a simulated driving task, it is 
very important to have the affective quality of the 
car's voice match the driver's own affective state 
(e.g., a subdued car voice when the driver is in a 
negative affective state, and a cheerful car voice 
when in a positive state). When these condi- 
tions were crossed, the drivers had twice as many 
accidents. One likely explanation is that the mis- 
matched condition required the driver to use more 
cognitive and attentive effort to interpret what the 
car was saying, distracting the driver from the 
changing road conditions. This study also points to 
the importance of enabling a computational system 
to sense and respond to a person's affect in real 
time: Adaptation to a person's emotion has a direct, 



measurable impact on safety. It is important to un- 
derstand how to design our interaction with tech- 
nology in ways that help us rather than hinder us. 
Whereas past effort in human-computer interac- 
tion has focused on the cognitive aspects, it is 
growing increasingly clear that it is also important 
to understand the affective aspects and how they 
influence cognitive processing. 



A New Scientific Tool 

While the potential applications (and consumer 
markets) for robotic companions demand substan- 
tial progress in this research area, it is also true that 
this applied research will advance basic scientific 
theories and understanding. The challenge in neu- 
roergonomics, to bring together new findings about 
the brain and cognition with new understanding of 
human behavior and human factors, requires ad- 
vances in tools. New tools are needed to sense 
emotion and behavior in natural environments and 
to provide controlled stimuli and responses in 
repeatable, measurable ways. While a natural inter- 
action between two people is rarely, if ever, repeat- 
able and controllable, and even scripted ones have 
a lot of variation (witness the differences in perfor- 
mance from evening to evening of two professional 
actors on stage in any long-running scripted pro- 
duction), the interactions made by a robot can be 
designed to be repeatable and measurable, quite 
precisely. 

The robot can be designed to sense and mea- 
sure a variety of channels of information from you 
(e.g., your eye and body movements associated 
with level and direction of attention) and can be 
built to respond in preprogrammed, precise ways 
(such as mirroring your postures). For example, La 
France's (1982) theory that mirroring of body pos- 
tures between people builds rapport and liking is 
based on observations of people by video. On the 
other hand, robots could be designed to respond in 
very precise and controlled ways to specific human 
states or behaviors, thus adding repeatability and a 
new measure of control in interaction studies. 
Hence, robots can be viewed not only as future 
companions for a variety of useful tasks, but also as 
research platforms for furthering basic scientific 
understanding of how emotion, cognition, and be- 
havior interact in natural and social situations. Our 
work involves answering a myriad of basic science 
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questions, as well as long-term goals of building 
companions that people would look forward to 
having around. 



Design of a Relational Robot 
for Education 

RoCo is our most recent effort in developing a ro- 
botic desktop computer with elements of social 
and emotional intelligence. Our primary motiva- 
tion for building a physically animated system is 
the development of applications that benefit from 
establishing a kind of social rapport between hu- 
man and computer. RoCo is an actuated version 
of a desktop computer where its monitor has the 
ability to physically move in subtly expressive 
ways that respond to its user (see figure 18.1). The 
movement of the machine is inspired by natural 
human-human interaction: When people work to- 
gether, they move in a variety of reciprocal ways, 
such as shifting posture at conversational bound- 
aries and leaning forward when interested (Argyle, 
1988). Physical movement plays an important role 
in aiding communication, building rapport, and 
even in facilitating the health of the human body, 
which was not designed to sit motionless for long 
periods of time. We have developed a suite of per- 



ceptual technologies that sense and interpret multi- 
modal affective cues from the user via custom sen- 
sors (including facial and postural expression 
sensing) using machine learning algorithms that 
have been designed to recognize patterns of human 
behavior from multiple modes. RoCo is designed 
to respond to the user's cues with carefully crafted 
subtle mechanical movements and occasional audi- 
tory feedback, using principles derived from natu- 
ral human-human interaction. 

The novelty of introducing ergonomic physical 
movement to human-computer interaction enables 
us to explore several interesting questions in the 
interaction of the human body with cognition and 
affect, and how this coupling could be applied to 
foster back health as well as task performance and 
learning gains of the user. For instance, one impor- 
tant open question we are exploring is whether the 
reciprocal movement of human and computer can 
be designed to promote back health, without being 
distracting or annoying to the user. Another in- 
triguing question is whether reciprocal physical 
movement of human and computer, and its inter- 
action with affect and cognition, could be designed 
to improve the humans efficacy of computer use. 
We believe this is possible in light of new theories 
that link physical posture and its influence on 
cognition and affect. An example is the theory in 





Figure 18.1. Concept drawing of the physically animated computer (left), and a mechanical prototype (right). 
RoCo can move its monitor "head" and its mechanical "neck" using 5 degrees of freedom. There are two axes of ro- 
tation at the head, an elbow joint, and a swivel and a lean degree of freedom at the base. It does not have explicit 
facial features, but this does not preclude the ability to display these graphically on the LCD screen. 
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the "stoop to conquer" research, where it was 
found that slumping following a failure in problem 
solving, and sitting up proudly following an achieve- 
ment, led to significantly better performance out- 
comes than crossing those conditions (Riskind, 
1984). 

Other theories, such as the role of subtle 
postural mirroring for building rapport and liking 
between humans (La France, 1982), could be ex- 
amined in human-computer interaction by using 
the proposed system to induce and measure subtle 
postural responses. In particular, nonverbal imme- 
diacy behaviors alone when displayed by a teacher 
(such as close conversational distance, direct body 
orientation, forward lean, and postural openness) 
have been shown to increase learning outcomes 
(Christensen & Menzel, 1998). Growing interest in 
the use of computers as pedagogical aids raises the 
question of the role of these postural immediacy 
behaviors in influencing the effectiveness of the 
learning experience provided by computers. It also 
motivates development of systems that can recog- 
nize and respond to the affective states of a human 
learner (e.g., interest, boredom, frustration, plea- 
sure, etc.) to keep the learner engaged and to help 
motivate the learner to persevere through chal- 
lenges in order to ultimately experience success 
(Aist, Kort, Reilly, Mostow, & Picard, 2002). 

RoCo as a Robotic Learning Companion 

Inspired by these scientific findings, we have been 
developing theory and tools that can be applied to 
RoCo so that it may serve as a robotic learning 
companion (Picard et al, 2004) that can help a 
child to persist and stay focused on a learning task, 
and mirrors some of the child's affective states to 
increase awareness of the role that these states play 
in propelling the learning experience. For example, 
if the child's face and posture show signs of intense 
interest in what is on the screen, this computer 
should hold very still so as to not distract the child. 
If the child shifts her posture and glances in a way 
that shows she is taking a break, the computer 
could do the same and may note that moment as 
a good time to interrupt the child and provide 
scaffolding (encouragement, tips, etc.) to help the 
learning progress. In doing so, the system would 
not only acknowledge the presence of the child and 
show respect for her level of attentiveness, but also 
would show subtle expressions that, in human- 



human interaction, are believed to help build rap- 
port and liking (La France, 1982). 

By increasing likeability, we aim to make the 
robotic computer more enjoyable to work with and 
potentially facilitate measurable task outcomes, 
such as how long the child perseveres with the 
learning task. Within K-6 education, there is evi- 
dence that relationships between students are im- 
portant in peer learning situations, including peer 
tutoring and peer collaborative learning method- 
ologies (Damon & Phelps, 1989). Collaborations 
between friends involved in these exercises have 
been shown to provide a more effective learning 
experience than collaboration between acquain- 
tances (Hartup, 1996). Friends have been shown 
to engage in more extensive discourse with one an- 
other during problem solving, to offer suggestions 
more readily, to be more supportive and more criti- 
cal, and to work longer on tasks and remember 
more about them afterward than nonfriends. 

Building Social Rapport 

In our experience, most computer agents in exis- 
tence are, at best, charming for a short interaction, 
but rapidly grow tiring or annoying with longer- 
term use. We believe that in order to build a robot 
or computer that people will enjoy interacting with 
over time, it must be able to establish and maintain 
a good social rapport with the user. This requires 
the robot to be able to accumulate memory of on- 
going interactions with the user and exhibit basic 
social-emotional skills. By doing so, it can respond 
in ways that appear socially intelligent given the 
user's affective state and expressions. Major progress 
has already been made within our groups in defin- 
ing, designing, and testing relational agents, com- 
puter and robotic agents capable of building 
long-term social-emotional relationships with peo- 
ple (Bickmore, 2003; Breazeal, 2002). 

We believe that relational and immediacy 
behaviors (see Richmond & McCroskey, 1995) — 
especially close proximity, direct orientation, anima- 
tion, and postural mirroring to demonstrate liking of 
the user and engagement in the interaction — should 
be expressible by a computer learning companion 
to help build rapport with the child. In support 
of this view, immediacy and relational behaviors 
have been implemented in a virtual exercise advi- 
sor application and shown to successfully build 
social rapport with the user (Bickmore, 2003). In a 
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99-person 1-month test of the exercise advisor 
agent, where subjects were split into three groups 
(one third with task but no agent, one third with 
task plus nonrelational agent, one third with task 
plus relational agent), we found task outcome al- 
ways improved, while a "bond" rating toward the 
agent was significantly higher (p < .05) in the 
relational case. This includes people reporting that 
the agent cared more about them, was more like- 
able, showed more respect, and earned more of 
their trust than the nonrelational agent. People 
interacting with the relational agent were also 
significantly more likely to want to continue inter- 
acting with that agent. Most of these measures 
were part of a standard instrument from clinical 
psychotherapy — the Working Alliance Inventory — 
that measures the trust and belief that the therapist 
and patient have in each other as team members in 
achieving a desired outcome (Horvath & Green- 
berg, 1989). The significant difference in peoples 
reports held at both times of evaluation: after 1 
week and after 4 weeks, showing that the improve- 
ment was sustained over time. These findings con- 
firm an important role for research in giving agents 
social-emotional and other relational skills for suc- 
cessful long-term interactions. 

Social Presence 

Why should the computer screen move, rather 
than simply moving a graphical character on a 
static screen? Does physical movement, especially 
drawing nearer or pulling further from the user, 
impact the social presence of a character? To an- 
swer this question, we carried out a series of 
human-robot interaction studies to explore how 
the medium through which an interaction takes 
place affects a person's perception of social pres- 
ence of a character (Kidd & Breazeal, 2004). In 
these studies, the robot had static facial features 
with movable eyes mounted upon a neck mech- 
anism (and therefore had degrees of freedom 
comparable to those proposed for our physically 
animate computer). The study involved naive sub- 
jects (n = 32) interacting with a physical robot, an 
on-screen animated character, and a person in a 
simple task. Each subject interacted with each of 
the characters, one at a time, in a preassigned or- 
der of presentation. The character made requests 
of the subject to move simple physical objects 
(three colored blocks). All requests were presented 



in a prerecorded female voice to minimize the ef- 
fects of different voices, and each character made 
these requests in a different order. At the conclu- 
sion of the interaction, subjects were asked to 
complete a questionnaire on their experiences 
based on the Lombard and Ditton scale for mea- 
suring social presence (Lombard et al., 2000). 
Subjects were asked to read and evaluate a series 
of statements and questions about engagement 
with the character on a seven-point scale. All data 
were evaluated using a single-factor ANOVA and 
paired two-sample t tests for comparisons between 
the robot and animated character. The data pre- 
sented were found to be statistically significant to 
p< .05. 

Our data show that the robot consistently 
scored higher on measures of social presence than 
the animated character (and both below that of the 
human). Overall, people found the robot character 
to be easier to read, more engaging of their senses 
and emotions, and more interested in them than 
the animated character. Subjects also rated the ro- 
bot as more convincing, compelling, and entertain- 
ing than the animated character. These findings 
suggest that in situations where a high level of mo- 
tivational and attentional arousal is desired, a phys- 
ically copresent and animated computer may be a 
preferred medium for the task over an animated 
character trapped within a screen. We are continu- 
ing this work by looking at more specific measures 
of the interaction (e.g., trust, reliability, and imme- 
diacy) when subjects interact with characters in 
different types of tasks, such as a cooperative task 
or a learning task. 



Sensing and Responding 
to Human Affect 

In this section, we briefly present three major com- 
ponents of our robotic computer system: 

• Expressive behavior of the robotic computer 

• Perceptual systems for passively sensing subtle 
movement and expression by the user 

• The cognitive-affective control system 

Expressive Behavior 

Character animators appreciate the importance of 
body posture and movement (i.e., the principle 
axes of movement) to convincingly portray life and 
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convey expression in inanimate objects (Thomas & 
Johnston, 1981). Hence, the primary use of the 
LCD screen for our purposes is to display task- 
relevant information, where physical movement is 
used for emotive and social expression. For in- 
stance, the mechanical expressions include pos- 
tural shifts like moving closer to the user, and 
looking around in a curious sort of way. However, 
it is possible to graphically render facial features on 
the screen if it makes sense to do so. 

We have developed expressive anthropomor- 
phic robots in the past that convey emotive states 
through facial expression and posture (Breazeal, 
2000). We have found that the scientific basis 
for how emotion correlates to facial expression or 
vocal expression is very useful in mapping the ro- 
bot's emotive states to its face actuators (Breazeal, 
2003a), and to its articulatory-based speech syn- 
thesizer (Breazeal, 2003b). In human- robot inter- 
action studies (Breazeal, 2003d), we have found 
that these expressive cues are effective in regulating 
affective or intersubjective interactions (Tre- 
varthen, 1979) and proto-dialogs (Tronick, Als, & 
Adamson, 1979) between the human and the robot 
that resemble their natural correlates during infant- 
caregiver exchanges. The same techniques are ap- 
plied to give RoCo its ability to express itself through 
posture. 

With respect to communicating emotion 
through the face, psychologists of the componen- 
tial theory of facial expression posit that these 
expressions have a systematic, coherent, and mean- 
ingful structure that can be mapped to affective 
dimensions that span the relationship between dif- 
ferent emotions (Smith & Scott, 1997). Some of 
the individual features of expression have inherent 
signal value. The raised brows, for instance, convey 
attentional activity for the expression of both fear 
and surprise. For RoCo, this same state can be 
communicated by an erect body posture. By con- 
sidering the individual facial action components 
that contribute to the overall facial display, it is 
possible to infer much about the underlying prop- 
erties of the emotion being expressed. This pro- 
motes a signaling system that is robust, flexible, 
and resilient. It allows for the mixing of these com- 
ponents to convey a wide range of affective mes- 
sages, instead of being restricted to a fixed pattern 
for each emotion. 

Inspired by this theory, RoCo's facial expres- 
sions and body postures can be generated using 



an interpolation-based technique over a three- 
dimensional affect space that we have used with 
past robots. The three dimensions correspond to 
arousal (high/low), valence (good/bad), and stance 
(advance/withdraw) — the same three attributes 
that our robots have used to affectively assess the 
myriad of environmental and internal factors that 
contribute to their overall affective state. There are 
nine basic postures that collectively span this space 
of emotive expressions. 

The current affective state of the robot (as 
defined by the net values of arousal, valence, and 
stance) occupies a single point in this space at 
a time. As the robot's affective state changes, this 
point moves around this space and the robot's fa- 
cial expression and body posture change to mirror 
this. As positive valence increases, the robot may 
move with a more buoyant quality. If facial expres- 
sions are shown on the screen, than lips would 
curve upward. However, as valence decreases, the 
posture would droop, conveying a heavy, sagging 
quality. If eyebrows are shown on the screen, they 
would furrow. Along the arousal dimension, the ro- 
bot moves quicker, with a more dartlike quality 
(positive), or with greater lethargy as arousal de- 
creases. Along the stance dimension, the robot 
leans toward (increasing) or shrinking away (de- 
creasing) from the user. These expressed move- 
ments become more intense as the affective state 
moves to more extreme values in the affect space. 

The computer can also express itself through 
auditory channels. The auditory expressions, de- 
signed to be similar in spirit to the fictional Star 
Wars robot R2D2, are nonlinguistic but aim to 
complement the movements, such as electronic 
sounds of surprise. This does not preclude the use 
of speech synthesis if the task demands it, but our 
initial emphasis is on using modes of communica- 
tion that do not explicitly evoke high-level human 
capabilities. 

Preliminary studies have been carried out to 
evaluate the readability of the physically animated 
computer's subtle expressions (Liu & Picard, 2003). 
In this preliminary study, 19 subjects watched 15 
different video clips of an animated version of the 
computer or heard audio sequences to convey cer- 
tain expressive behaviors (e.g., welcoming, sorrow, 
curious, confused, and surprised). The expressive 
behaviors were conveyed through body movement 
only — the LCD screen was blank. The subjects 
were asked to rate the strength of each of these 
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expressions on a seven-point scale for video only, 
audio only, or video with audio. In a two-tailed 
t test, there was significant recognition of the be- 
havior sequences designed to express curiosity, 
sadness, and surprise. These initial studies are en- 
couraging, and we are continuing to refine expres- 
sive movements and sounds. 

Perceptual Systems 

We are interested in sensing those affective and at- 
tentive states that play an important role in ex- 
tended learning tasks. Detecting affective states 
such as interest, boredom, confusion, and excite- 
ment are important. Our goal is to sense these 
emotional and cognitive aspects in an unobtrusive 
way. Cues like the learner's posture, gesture, eye 
gaze, facial expression, and so on help expert 
teachers recognize whether a learner is on-task or 
off-task. For instance, Rich et al. (1994) have de- 
fined symbolic postures that convey a specific 
meaning about the actions of a user sitting in an of- 
fice, such as interested, bored, thinking, seated, re- 
laxed, defensive, and confident. Leaning forward 
toward a computer screen might be a sign of atten- 
tion (on-task), while slumping on the chair or fidg- 
eting suggests frustration or boredom (off- task). 

We have developed a postural sensing system 
with custom pattern recognition that watches the 
posture of children engaged in computer learning 
tasks and learns associations between their pos- 
tural movements and their level of interest — high, 



low, or "taking a break" (a state of shifting forward 
and backward, sometimes with hands stretched 
above the head, which tended to occur frequently 
before teachers labeled a child as bored). The sys- 
tem attained 82% recognition accuracy training 
and testing on different episodes within a group of 
eight children, and performed at 77% accuracy 
recognizing these states in two children that the 
system had not seen before (Mota, 2002; Mota & 
Picard, 2003). 

A computer might thus use postural cues to 
decide whether it is a good time to encourage the 
user to take a break and stretch or move around. 
Specifically, we have focused on identifying the 
surface-level behaviors (indicative of both attention 
and affect) that suggest a transition from an on-goal 
to off-goal state or vice versa. Toward this aim, we 
are further developing and integrating multimodal 
perceptual systems that allow RoCo to extract this 
information about the user. The systems under de- 
velopment are described in the following sections. 

The Sensor Chair 

The user sits in a sensor chair that provides 
posture data from an array of force-sensitive 
resistors — similar to the Smart Chair used by Tan, 
Ifung, and Pentland (1997). It consists of two 
0.10 mm thick sensor sheets, with an array of 
42 x 48 sensing units. Each unit outputs an 8-bit 
pressure reading. One of the sheets is placed on 
the backrest and one on the seat (see figure 18.2). 
The pressure distribution map (2 of 42 x 48 




Figure 18.2. The Smart Chair (left) and data (right) showing characteristic pressure patterns used to recognize 
postural shifts characteristic of high interest, low interest, and taking a break. See also color insert. 
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Figure 18.3. A snapshot of the stereo vision system that is mounted in the base of the computer (devel- 
oped in collaboration with the Vision Interfaces Group at MIT CSAIL). Motion is detected in the upper 
left frame; human skin chromaticity is extracted in the lower left frame; a foreground depth map is com- 
puted in the lower right frame; and the faces and hands of audience participants are tracked in the upper 
right frame. See also color insert. 



points) sensed at a sampling frequency of 50 Hz. 
A custom pattern recognition system was devel- 
oped to distinguish a set of 9 static postures that 
occur frequently during computer learning tasks 
(e.g., lean forward, lean back, etc.), and to analyze 
patterns of these postures over time in order to 
distinguish affective states of high interest, low in- 
terest, and taking a break. These postures and 
associated affective states were recognized with 
significantly greater than random accuracy indi- 
cating that the postural pressure cues carry signifi- 
cant information related to the student's interest 
level (Mota & Picard, 2003). 

Color Stereo Vision 

A color Mega-D stereo vision system (manufac- 
tured by Small Vision Systems, Inc.) is mounted 
inside the computers base. This is a compact, 
integrated megapixel system that uses advanced 



CMOS imagers from PixelCam and fast 1394 bus 
interface electronics from Vitana to deliver high- 
resolution, real-time stereo imagery to any PC 
equipped with a 1394 interface card. We use it to 
detect and track the movement and orientation of 
the user's hands and face (Breazeal et al, 2003; 
Darrell, Gordon, & Woodfill, 1998). This vision 
system segments people from the background and 
tracks the face and hands of the user who interacts 
with it. We have implemented relatively cheap al- 
gorithms for performing certain kinds of model- 
free visual feature extraction. A stereo correlation 
engine compares the two images for stereo corre- 
spondence, computing a 3-D depth (i.e., disparity 
map) at about 15 frames per second. This is com- 
pared with a background depth estimate to pro- 
duce a foreground depth map. The color images 
are simultaneously normalized and analyzed with a 
probabilistic model of human skin chromaticity to 
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segment out areas of likely correspondence to hu- 
man flesh. The foreground depth map and the skin 
probability map are then filtered and combined, 
and positive regions extracted. An optimal bound- 
ing ellipse is computed for each region. For the 
camera behind the machine facing the user, a 
Viola-Jones face detector (Viola & Jones, 2001) 
runs on each to determine whether or not the re- 
gion corresponds to a face. The regions are then 
tracked over time, based on their position, size, 
orientation, and velocity. Connected components 
are examined to match hands and faces to a single 
owner (see figure 18.3). 

Blue Eyes Camera 

An IBM Blue Eyes camera system is mounted on 
the lower edge of the LCD screen (http://www 
.almaden.ibm.com/cs/blueeyes), as shown in figure 
18.4. It uses a combination of off-axis and on-axis 
infrared LEDs and an infrared camera to track the 
user's pupils unobtrusively by producing the red 
eye effect (Haro, Essa, & Flickner, 2000). We have 
developed real-time techniques using the pupil 
tracking system to automatically detect the users 
facial features (e.g., eyes, brows, etc.), gaze direc- 
tion, and head gestures such shaking or nodding 
(Kapoor & Picard, 2002; Kapoor, Qi, & Picard, 
2003). Also physiological parameters like pupillary 
dilation, eye-blink rate, and so on can be extracted 
to infer information about arousal and cognitive 
load. The direction of eye gaze is an important sig- 
nal to assess the focus of attention of the learner. In 
an on-task state, the focus of attention is mainly to- 
ward the problem the student is working on, 
whereas in an off-task state the eye gaze might 
wander off from it. 

Mouse Pressure Sensor 

We have modified a computer mouse to sense not 
only the usual information (where the mouse is 
placed and when it is clicked) but also how it is 
clicked, the adverbs of its use (see figure 18.5). It 
has been observed that users apply significantly 
more pressure to the mouse when a task is frustrat- 
ing to them than when it is not (Dennerlein, 
Becker, Johnson, Reynolds, & Picard, 2003). Use 
of this sensor is new and much is still to be learned 
about how its signals change during normal use. 
We propose to replace the traditional computer 
mouse in our setup with this pressure mouse, and 



combine its data with that from the other sensors, 
to assess if it is also helpful in learning about the 
user state. 

In addition to the physical synchronization 
of data from the sensors described above, we 
have also developed algorithms that integrate these 
multiple channels of data for making joint infer- 
ence about the human state. Our approach refines 
recent techniques in machine learning based on a 
Bayesian combination of experts. These techniques 
use statistical machine learning to learn multiple 
classifiers and to learn how they tend to perform 
given certain observations. This information is 
then combined to learn how to best combine their 
outputs to make a joint decision. The techniques 
we have been developing here generalize those of 
Miller and Yan (1999) and so far appear to perform 
better than classifier combination methods such as 
the product rule, sum rule, vote, max, min, and so 
forth (Kapoor et al., 2003). We are continuing to 
improve upon these methods using better approxi- 
mation techniques for the critics and their perfor- 
mance, as well as integrating new techniques that 
combine learning from both labeled and unlabeled 
data (semisupervised, with the human in the loop). 

Cognitive-Affective Control System 

Robotics researchers continue to be inspired by sci- 
entific findings that reveal the reciprocally interre- 
lated roles that cognition and emotion play in 
intelligent decision making, planning, learning, at- 
tention, communication, social interaction, mem- 
ory, and more (Lewis & Haviland-Jones, 2000). 
Several early works in developing computational 
models of emotions for robots were inspired by 
theories of basic emotions and the role they play in 
promoting a creatures survival in unpredictable 
environments given limited resources (Ekman, 
1992; Izard, 1977). Later works explored different 
models (such as cognitive appraisal models) and 
their theorized role in intelligent decision making, 
while other works have explored the role of affect 
in reinforcement learning (see Picard, 1997, for a 
review). 

Our own work has explored various social- 
emotional factors that benefit a robot's ability to 
achieve its goals (Breazeal, 2002) and to learn new 
skills (Lockerd & Breazeal, 2004) in partnership 
with people. For instance, the design of our first 
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Figure 18.4. The Blue Eyes camera (left) and data (right) showing its ability to detect the user's pupils. 



socially interactive robot, called Kismet, was in- 
spired by mother- infant communicative interactions. 
Any parent can attest to the power that infants' 
social-emotional responses have in luring adults to 
help satisfy their infant's needs and goals. Infants 
socially shape their parents' behavior by exhibiting 
a repertoire of appropriate negative responses to 
undesirable or inappropriate interactions (e.g., fear, 
disgust, boredom, frustration) and a repertoire of 
positive responses (e.g., happiness, interest) to 
those that help satisfy their goals. Similarly, Kismet's 
emotive responses were designed and have been 
demonstrated in human-robot interaction studies to 
do the same (Breazeal, 2002; Turkle, Breazeal, De- 
tast, & Scassellati, 2004). 

Our cognitive-affective architecture features 
both a cognitive system and an emotion system 
(see figure 18.6). It is being constructed using the 
C5 behavior architecture system developed at the 
MIT Media Lab (Burke, Isla, Downie, Ivanov, & 
Blumberg, 2001) to create autonomous, animated, 
interactive characters that interact with each other 
as well as the user. Whereas the cognitive system is 
responsible for interpreting and making sense of 
the world, the emotion system is responsible for 
evaluating and judging events to assess their over- 
all value with respect to the creature, for example, 
positive or negative, desirable or undesirable, and 
so on. These parallel systems work in concert to 
address the competing goals and motives of the ro- 
bot given its limited resources and the ever- 
changing demands of interacting with people. 

The cognitive system (the light-gray modules 
shown in figure 18.6) is responsible for perceiving 



and interpreting events, and for arbitrating among 
the robot's goal-achieving behaviors to address 
competing motivations. To achieve this, we based 
the design of the cognitive system on classic etho- 
logical models of animal behavior (heavily inspired 
by those models proposed by Tinbergen [1951] 
and Lorenz [1950]). The computational subsystems 
and mechanisms that comprise the cognitive system 
(i.e., perceptual releasers, visual attention, homeo- 
static drives, goal-oriented behavior and arbitra- 
tion, and motor acts) work in concert to decide 
which behavior to activate, at what time, and for 
how long to service the appropriate objective. Over- 
all, the robot's behavior must exhibit an appropri- 
ate degree of relevance, persistence, flexibility, and 
robustness. 

The robot's emotion system implements the 
style and personality of the robot, encoding and 
conveying its attitudes and behavioral inclina- 
tions toward the events it encounters. Further- 
more, these cognitive mechanisms are enhanced 
by emotion-inspired mechanisms (the white mod- 
ules shown in figure 18.6), informed by basic 
emotion theory, that further improve the robot's 
communicative effectiveness via emotively expres- 
sive behavior, its ability to focus its attention on 
relevant stimuli despite distractions, and its ability 
to prioritize goals to promote flexible behavior 
that is suitably opportunistic when it can afford 
to be, yet persistent when it needs to be. The emo- 
tion system achieves this by assessing and signal- 
ing the value of immediate events in order to 
appropriately regulate and bias the cognitive sys- 
tem to help focus attention, prioritize goals, and to 
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Figure 18.5. Pressure-sensitive mouse collects data on how you handle it — the waveform is higher when more 
pressure is applied. 



pursue the current goal with an appropriate degree 
of persistence and opportunism. Furthermore, the 
emotive responses protect the robot from intense 
interactions that may be potentially harmful, and 
help the robot to sustain interactions that are bene- 
ficial for the robot. 

Broadly speaking, each emotive response con- 
sists of the four factors described in the following 
sections. 

A Precipitating Event 

Precipitating events are detected by perceptual pro- 
cesses. Environmental stimuli influence the robot's 
behavior through perceptual releasers (inspired by 
Tinbergen's innate releasing mechanisms). Each re- 
leaser is modeled as a simple elicitor of behavior 
that combines lower-level perceptual features into 
behaviorally significant perceptual categories. 

An Affective Appraisal of That Event 

Behavioral responses can be pragmatically elicited 
by rewards and punishments, where a reward is 
something for which an animal (or robot) will 
work and a punishment is something it will work 
to escape or avoid. The robots affective appraisal 
process assesses whether an internal or external 
event is rewarding, punishing, or not relevant, and 
tags it with an affective value that reflects its ex- 
pected benefit or harm to the robot. This mecha- 
nism is inspired by Damasio's somatic marker 
hypothesis (Damasio, 1994). For instance, internal 
and external events are appraised in relation to the 
quality of the stimulus (e.g., the intensity is too 
low, too high, or just right), or whether it relates to 
the robot's current goals or motivations. 

There are three classes of tags used within the 
robot to affectively characterize a given event. 
Each tag has an associated intensity that scales its 



contribution to the overall affective state. The 
arousal tag (A) specifies how arousing (or intense) 
this factor is and very roughly corresponds to the 
activity of the autonomic nervous system. Positive 
values correspond to high arousal whereas nega- 
tive values correspond to low arousal. The valence 
tag (V) specifies how favorable or unfavorable the 
event is to the robot and varies with the robot's 
current goals. Positive values correspond to a ben- 
eficial (desirable) event, whereas negative values 
correspond to an event that is not beneficial (not 
desirable). The stance tag (S) specifies how ap- 
proachable the event is — positive values encour- 
age the robot to approach whereas negative values 
correspond to retreat. It also varies with the robot's 
current goals. 

These affective tags serve as the common cur- 
rency for the inputs to the behavioral response 
selection mechanism, allowing the influences of 
externally elicited perceptual states and a rich 
repertoire of internal states to be combined. Per- 
ceptual states contribute to the net affective state of 
the robot based on their relevance to the robot's 
current goals or their intrinsic affective value ac- 
cording to the robot's programmed preferences. 
The ability of the robot to keep its drives satiated 
also influences the robot's affective state: positive 
when a drive is in balance, and negative when out 
of balance. The ability of the robot to achieve its 
goals is another factor. Forward progress culminat- 
ing in success is tagged with positive valence. In 
contrast, prolonged delay in achieving a goal is 
tagged with negative valence. 

A Characteristic Display 

Characteristic display can be expressed through fa- 
cial expression, vocal quality, or body posture, as 
described earlier. 
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Figure 18.6. An overview of the cognitive-affective architecture showing the tight integration of the cognitive sys- 
tem (shown in light gray), the emotion system (shown in white), and the motor system. The cognitive system com- 
prises perception, attention, drives, and behavior systems. The emotion system comprises affective releasers, 
appraisals, elicitors, and gateway processes that orchestrate emotive responses. A, arousal tag; V, valence tag; 
S, stance tag. 
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Modulation of the Cognitive and Motor Systems 
to Motivate a Behavioral Response 

The emotive elicitor system combines the myriad 
of affectively tagged inputs (i.e., perceptual elici- 
tors, cognitive drives, behavioral progress, and 
other internal states) to compute the net affective 
state of the robot. This affective state characterizes 
whether the robot's homeostatic needs are being 
met in a timely manner, and whether the present 
interaction is beneficial to its current goals. When 
this is the case, the robot is in a mildly positive, 
aroused, and open state, and its behavior is pleas- 
ant and engaging. When the robot's internal state 
diverges from this desired internal relationship, it 
will work to restore the balance — to acquire de- 
sired stimuli, to avoid undesired stimuli, and to es- 
cape dangerous stimuli. Each emotive response 
carries this out in a distinct fashion by modulating 
the cognitive system to redirect attention, to evoke 
a corrective behavioral pattern, to emotively com- 
municate that the interaction is not appropriate, 
and to socially cue others how they might respond 
to correct the problem (e.g., a fearful response to a 
threatening stimulus, an angry response to some- 
thing blocking the robot's goal, a disgusted re- 
sponse to a undesirable stimuli, a joyful response 
to achieving its goal, etc.). In a process of behav- 
ioral homeostasis, the emotive response maintains 
activity through external and internal feedback un- 
til the correct relation of robot to environment is 
established (Plutchik, 1991). 



Social Learning 



well as robots), social referencing is extremely 
useful in forming early appraisals and coping re- 
sponses toward unfamiliar stimuli with the help of 
others. 

Referencing behavior operates primarily un- 
der conditions of uncertainty — if the situation has 
low ambiguity, then intrinsic appraisal processes 
are used (Campos & Stenberg, 1981). In particu- 
lar, emotional referencing is viewed as a process 
of emotional communication whereby the infant 
learns how to feel about a given situation, and 
then responds to the situation based on his or her 
emotional state (Feinman, Roberts, Hsieh, Sawyer, 
& Swanson, 1992). For example, the infant might 
approach a toy and kiss it upon receiving a joy 
message from the adult, or swat the toy aside 
upon receiving a fear message (Hornik & Gunnar, 
1988). 

In our model, shown in figure 18.7 (for our ex- 
pressive humanoid robot, Leonardo), a perception- 
production coupling of the robot's facial imitation 
abilities based on the active-intermodal mapping 
hypothesis of Meltzoff and Moore (1997) is lever- 
aged to allow the robot to make simple inferences 
about the emotional state of others. Namely, imitat- 
ing the expression of another as they look upon the 
novel object induces the appropriate affective state 
within the robot. The robot applies this induced 
affective state as the appraisal for the novel object 
via the robot's joint attention and emotion-based 
mechanisms. This allows Leonardo to use the emo- 
tionally communicated assessment of others to form 
its own appraisals of the same situations and to 
guide its own subsequent responses. 



We have been adapting this architecture to ex- 
plore various forms of social learning on robots, 
such as tutelage (Lockerd & Breazeal, 2004), imi- 
tation (Buschbaum, Blumberg, & Breazeal, 2004), 
and social referencing (Breazeal, Buchsbaum, Gray, 
Gatenby, & Blumberg, 2004). For instance, social 
referencing is an important form of socially 
guided learning in which one person utilizes an- 
other person's interpretation of a given situation to 
formulate his or her own interpretation of it and 
to determine how to interact with it (Feinman, 
1982; Klinnert, Campos, Source, Emde, & Svejda, 
1983). Given the large number of novel situa- 
tions, objects, or people that infants encounter (as 



Conclusion 

This chapter summarizes our ongoing work in de- 
veloping and embedding affective technologies in 
learning interactions with automated systems such 
robotic learning companions. These technologies 
include a broad repertoire of perceptual, expres- 
sive, and behavioral capabilities. Importantly, we 
are also beginning to develop computational mod- 
els and learning systems that interact with people 
to elucidate the role that social, cognitive, and af- 
fective mechanisms play in learning (see Picard et 
al., 2004, for a review). By doing so, we hope to 
better answer such questions as: What affective 
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Visual Perception 

1. Leo is shown an unfamiliar Object_X. 

7. Leo sees human and her expression. 
She is looking at object and smiling. 

13. Leo looks back at object to share 
attention with human. 






Feature Attributes 



Attention & 3D Spatial Model 

2. Robot Attention Focus: Object_X 

8. Robot Attention Focus: Face 
Human Attention Focus: Object_X 

14. Referential Focus: Object_X 
Robot Attention Focus: Object_X 
Human Attention Focus: Object_X 



Attentional Focus 
Referential Focus 
^/^y^ 1 / P erce P c Attributes 



□ First Pass 

□ Second Pass 

□ Third Pass 



Belief System & 
Affective Appraisal 

3. New belief created: Object_X 
Robot Attention Focus: Object_X 
Affective Tag: Novel 

9. Human belief updated: 
Robot Attention Focus: Face 
Human Attention Focus: Object_X 
Expression: Smile 
Affective Tag: Positive 

15. Update belief Object_X: 
Robot Attention Focus: Object_X 
Human Attention Focus: Object_X 
Affective Tag: Positive 



Emotion System 



J\ 4. Uncertainly triggers anxious emotional state. 
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10. Person and expression triggers positive 
emotion state. 

16. Positive emotion activates engagement of 
toy with positive appraisal. 



Releasers 
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Action System 



5. Anxious emotion activates behavior to find 
human to seek information and reduce 
uncertainty. 

1 1 . Positive emotion and looking at human's 
face activates engagement and interaction 
behaviors to share attention. 

17. Positive emotive state elicited by positive 
appraisal of toy activates engagement behavior. 



Motor System 

6. Look at human. 

12. Look to novel toy to share attention. , \r- 

18. Reach for toy. 
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Figure 18.7. Model of social referencing. This schematic shows how social referencing is implemented within 
Leonardo's extended cognitive-affective architecture. Significant additions include a perception-production system 
for facial imitation, an attention system that models the attentive and referential state of the human and the robot, 
and a belief system that bundles visual features with attentional state to represent coherent entities in the 3-D space 
around the robot. The social referencing behavior executes in three passes through the architecture, each pass 
shown by a different colored band. The numbers represent steps in processing as information flows through the ar- 
chitecture. In the first pass, the robot encounters a novel object. In the second pass, the robot references the hu- 
man to see his or her reaction to the novel object. On the third pass, the robot uses the human's assessment as a 
basis to form its own affective appraisal of the object (step 15) and interacts with the object accordingly (step 18). 



states are most important to learning and how do 
these states change with various kinds of peda- 
gogy? How does knowledge of one's affective state 
influence outcomes in the learning experience? Ad- 
ditionally, these technologies form the basis for 
building systems that will interact with learners in 
more natural ways, even bootstrapping the ma- 
chine's own ability to learn from people. 

In the grand picture, we hope to realize three 
equally important goals. First, we wish to advance 
the state of the art in machine learning to develop 
systems that can learn far more quickly, broadly, 
and continuously from natural human instruction 
and interaction than they could alone. The ability of 



personal service robots of the future to quickly 
learn new skills from humans while on the job will 
be important for their success. Second, we aspire to 
achieve a deeper understanding of human learning 
and development by creating integrated models 
that permit an in-depth investigation into the so- 
cial, emotional, behavioral, and cognitive factors 
that play an important role in human learning. 
Third, we want to use these models and insights to 
create engaging technologies that help people learn 
better. We believe that such models can provide 
new insights into numerous cognitive-affective 
mechanisms and shape the design of learning tools 
and environments, even if they do not compare to 
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the marvelous nature of those that make children 
tick. 



Main Points 

1. Endowing robots with social-emotional skills 
will be important where robots are expected to 
interact with people, and will be especially 
critical when the interactions involve 
supporting various requests and needs of 
humans. 

2. New tools are needed, and are under 
development, for real-time multimodal 
perception of human affective states in natural 
environments and for providing controlled 
stimuli in repeatable and measurable ways. 
This requires new technological developments 
to create a broad repertoire of perceptual, 
expressive, and behavioral capabilities. It also 
requires the development of computational 
models to elucidate the role that physical, 
social, cognitive, and affective mechanisms 
play in intelligent behavior. 

3. We are developing a physically animate 
desktop computer system (called RoCo) to 
explore the interaction of the human body 
with cognition and affect, and how this 
coupling could be applied to foster back 
health as well as task performance and 
learning gains for the human user. It also 
enables us to explore scientific questions such 
as "What affective states are most important to 
learning?" "How awareness of one's own 
affective states might influence learning 
outcomes?" and so on. 

4. Successful human-computer interaction over 
the long term remains a significant challenge. 
The ability for a computer to build and 
maintain social rapport with its user (via 
immediacy behaviors and conveying liking or 
caring) could not only make computers more 
engaging, but also provide performance 
benefits for the human user by establishing an 
effective working alliance for behavior change 
goals or learning gains. 
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Neural Engineering 



The desire to re-create or at least emulate the func- 
tions of the nervous system has been one of the 
prime movers of computer science (Von Neumann, 
1958) and artificial intelligence (Marr, 1977). In 
pursuit of this objective, the interaction between 
engineering and basic research in neuroscience 
has produced important advances in both camps. 
Key examples include the development of artificial 
neural networks for automatic pattern recognition 
(Bishop, 1996) and the creation of the field of 
computational neuroscience (Sejnowski, Koch, & 
Churchland, 1988), which investigates the mecha- 
nisms for information processing in the nervous 
system. At the beginning of this new millennium, 
neural engineering has emerged as a rapidly grow- 
ing research field from the marriage of engineering 
and neuroscience. 

As this is a new discipline, a firm definition of 
neural engineering is not yet available. And per- 
haps it would not be a good idea to constrain the 
growing entity within narrow boundaries. Yet a 
distinctive focus of neural engineering appears to 
be in the establishment of direct interactions be- 
tween the nervous system and artificial devices. As 
pointed out by Buettener (1995), this focus leads 
to a partition of the field into two reciprocal appli- 
cation domains: the application of technology to 



improve biological function and the application of 
biological function to improve technology. 

In the first case, the physical connection be- 
tween neural and artificial systems aims primarily 
at restoring functions lost to disabling conditions, 
such as brain injury, stroke, and a large number 
of chronic impairments. In the second case, one 
can visualize the visionary goal of tapping into 
the as-yet unparalleled computational power of 
the biological brain and of its constituents to 
build intelligent, adaptable, and self-replicating 
machines. 

Both cases involve the combination of science 
and technology from fields as diverse as cellular 
and molecular biology, material science, signal 
processing, artificial intelligence, neurology, reha- 
bilitation medicine, and robotics. But perhaps 
most remarkably, in both cases the focus on estab- 
lishing information exchange between artificial 
and neural systems is likely to lead to deeper ad- 
vances of our basic understanding of the nervous 
system itself. 

While neural engineering appears to be a new 
field, emerging at the beginning of the new millen- 
nium, its roots reach quite far into the past. This 
chapter begins with a brief review of the earlier 
ideas and advances in the interaction between 
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neural and artificial systems. A significant impulse 
to this field is contributed by the need to develop 
artificial spare parts for our bodies, such as arms 
and legs. In a subsequent section, we present ideas 
concerning the interaction between design and 
neural control of such prosthetic devices. The per- 
spective of using neural control for artificial devices 
has been enhanced by advances in our ability to ex- 
tract information from brain signals. We then re- 
view recent advances in the use of signals extracted 
by electroencephalography (EEG) and by electrodes 
implanted in cortical areas for generating com- 
mands to artificial devices. Subsequently, we dis- 
cuss the role of sensory feedback and the problem 
of creating an artificial version of it by transmitting 
sensory information to the nervous system via elec- 
trical stimulation. The next section offers a view on 
the clinical impact of neural engineering, both in its 
current state and in the foreseeable future. 

A second perspective on neural engineering 
concerns the development of novel technology 
based on the computational power of the nervous 
system. One issue associated with this perspective 
is the development of biomimetic and hybrid tech- 
nologies, which we consider in this chapter. Oth- 
ers, considered in subsequent sections, include 
the analysis of neural plasticity as a biological pro- 
gramming language and the use of interactions be- 
tween brain and machines as a means to investigate 
neural computation. 



From Fiction to Reality: 
Cochlear Implants 

The marriage of biological and artificial systems 
has captured the dreams and nightmares of hu- 
manity since biblical times. One of the earliest 
ideas in this regard is the golem, the automaton 
made of clay first mentioned in the Talmud, resur- 
facing in the 16th century's legend of a creature 
created by Rabbi Loews of Prague to protect and 
support the city's Jewish community. However, not 
everything worked as planned, because the golem 
became a threat to innocent lives and the rabbi had 
to deprive it of its spiritual force. Interfering with 
nature in such potentially hazardous ways has been 
of great concern to literary figures since at least the 
19th century. We see these themes emerge in Mary 
Shelley's Frankenstein and in the reappearance of 



the golem in the work of Isaac Singer and in the 
classic film by Paul Wegener, on the eve of World 
War II. 

Years later, Manfred Clynes and Nathan Kline 
(1960) coined the term cyborg to define the inti- 
mate interaction of human and machine. This term 
originated in the semifictional context of space 
exploration, from the idea of augmenting human 
sensory and motor capabilities through artificial 
motion and sensing organs. As in the case of the 
golem and Frankenstein, cyborgs rapidly became 
part of a fearful view of a future in which we create 
monstrous creatures that threaten our own exis- 
tence. This rather dark view was particularly in- 
tense at a time of global wars, when millions of 
lives were destroyed. And, indeed, all technologies 
are potential sources of mortal danger. However, 
one can look at them with a different perspective 
(Clark, 2003). This is particularly the case for 
human-machine interactions, which are contribut- 
ing in a variety of ways to enhance the quality of 
life of the disabled population. 

Cochlear implants (Loeb, 1990; Rubinstein, 
2004) are among the earliest and most successful 
interfaces between brain and artificial systems 
(figure 19.1). Currently, some 40,000 patients 
worldwide have such implants, which allow deaf 
adult recipients to speak on the phone and chil- 
dren to attend regular school classes (Rauschecker 
& Shannon, 2002). Cochlear implants are based 
upon the simple concept of transforming the signals 
picked up by a microphone into electrical stimula- 
tions, which are delivered to the basilar membrane 
of the cochlea. 

In normal conditions, the cochlea transforms 
the sound vibrations generated by the eardrums 
into neural impulses that are transmitted via the 
acoustic nerve to the cochlear nucleus of the brain 
stem and then to the auditory cortex. An important 
characteristic of this signal transduction is the sep- 
aration of spectral components that are topograph- 
ically organized in regions sensitive to specific 
frequency bands (Merzenich & Brugge, 1973). A 
key step in the development of cochlear implants 
has been the understanding of the critical value of 
this spectral organization or tonotopy (Loeb, 1990; 
Rauschecker & Shannon, 2002), which has led 
the pioneers of this technology to decompose 
the acoustic signal into bands of increasing me- 
dian frequency and to deliver the corresponding 
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Figure 19.1. Cochlear implant. From Rauschecker and Shannon (2002). See 
also color insert. 



electrical stimuli to contiguous portions of the 
cochlear membrane. 

While these stimuli bear some resemblance 
to natural sound stimuli, patients typically do not 
reacquire new hearing ability right after the implant. 
At first, they most often hear only noise. The recov- 
ery of hearing involves a reorganization of neural in- 
formation processing, particularly in the auditory 
cortex. Reorganization may take place thanks to the 
plastic properties of the nervous system. 

Interestingly, this learning and reorganization 
process has highlighted a significant difference be- 
tween children that receive the implant before the 
onset of language and children that receive it after- 
ward. The latter group shows a much faster adapta- 
tion to the implant and greater ability to extract 



sound-related information (Balkany, Hodges, & 
Goodman, 1996; Rauschecker, 1999). However, 
there is also a trend toward the reduction of plas- 
ticity in the auditory and visual system as devel- 
opment progresses toward adulthood. Therefore, 
younger postlingual patients have generally better 
results than older adults. The possibility of success 
with cochlear implants is contingent upon the con- 
ditions of the cochlea and of the pathways joining 
the cochlea to the central auditory system. These 
conditions are compromised in pathological condi- 
tions such as type 2 neurofibromatosis, which is 
treated by surgical transection of the auditory 
nerve and causes total deafness. 

Auditory-brain stem interfaces work on 
the same signal-processing principles as cochlear 
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implants, but the stimulating electrodes are placed 
in the brain stem, over the cochlear nucleus. This 
offers a potential alternative when the peripheral 
auditory system is compromised or disconnected 
from the central auditory system. The outcomes of 
these interfaces are promising (Kanowitz, 2004), 
but not yet as impressive as those of cochlear im- 
plants. One of the possible causes is the less acces- 
sible distribution of frequencies inside the cochlear 
nucleus compared to the tonotopic organization of 
the cochlea. To address this issue, researchers are 
developing electrodes that deliver at different 
depths signals encoding different frequencies 
(McCreery, Shannon, Moore, & Chatterjee, 1998). 
Success in dealing with the transmission of sen- 
sory information to central brain structures will 
have critical implications for the use of artificial 
electrical stimuli as a source of feedback to the 
motor system. 



Current Challenges in 
Motor Prosthetics 

There have been many attempts to design fully ar- 
ticulated arms and hands in an effort to re-create 
the full function of the hand. While the ultimate 
goal of upper-extremity prosthetics research is the 
meaningful subconscious control of a multifunc- 
tional prosthetic arm or hand — a true replacement 
for the lost limb — the current state-of-the-art elec- 
tric prosthetic hands are generally single degree-of- 
freedom (opening and closing) devices usually 
implemented with myoelectric control (electric sig- 
nals generated as a by-product of normal muscle 
contraction). Current prosthetic arms requiring 
multiple-degree-of-freedom control most often use 
sequential control. Locking mechanisms or special 
switch signals are used to change control from one 
degree of freedom to another. As currently imple- 
mented, sequential control of multiple motions is 
slow; consequently, transradial prostheses are gen- 
erally limited to just opening and closing of the 
hand, greatly limiting the function of these devices 
(figure 19.2). Persons with recent hand amputa- 
tions expect modern hand prostheses to be like 
hands, similar to depictions of artificial hands in 
stories like The Six Million Dollar Man or Star Wars. 
Because these devices fail to meet some users' ex- 
pectations, they are frequently rejected (Weir, 
2003). 



A major factor limiting the development of 
more sophisticated hand and arm prostheses is the 
difficulty of finding sufficient control sources to 
control the many degrees of freedom required to 
replace a physiological hand or arm. In addition, 
most multiple-degree-of-freedom prosthetic hands 
are doomed by practicality, even before the control 
interface becomes an issue. Most mechanisms fail 
because of poor durability, lack of performance, 
and complicated control. No device will be clini- 
cally successful if it breaks down frequently. A mul- 
tifunctional design is by its nature more complex 
than a single-degree-of-freedom counterpart. How- 
ever, some robustness and simplicity must be 
traded if the increase in performance possible with 
a multiple-degree-of-freedom hand is ever to be re- 
alized. 

Neuroelectric control in its broadest context 
arises where a descending neural command is 
readily interpreted by a peripheral apparatus, and 
an ascending command returned, holds the allure 
of being able to provide multiple-channel control 
and multiple-channel sensing. There has been much 
research into interfacing prosthesis connections 
directly to nerves and neurons (Andrews et al., 
2001; Edell, 1986; Horch, 2005; Kovacs, Storment, 
James, Hentz, & Rosen, 1988) but the practicality 
of human-machine interconnections of this kind 
is still problematic. Nervous tissue is sensitive to 
mechanical stresses, and sectioned nerves are 
more sensitive still, in addition to which, this 
form of control also requires the use of implanted 
systems. 

Edell (1986) attempted to use nerve cuffs to 
generate motor control signals. Kovacs et al. 
(1988) tried to encourage nerve fibers to grow 
through arrays of holes in silicon integrated circuits 
that had been coated with nerve growth factor. An- 
drews et al. (2001) reported on their progress in 
developing a multipoint microelectrode peripheral 
nerve implant. Horch (2005) has demonstrated the 
control of a prosthetic arm with sensory feedback 
using needle electrodes in peripheral nerves, but 
these are short-term experiments with the elec- 
trodes being removed after 3 weeks. 

The development of BIONs (Loeb et al., 1998) 
for functional electrical stimulation (FES) is a 
promising new implant technology that may have a 
far more immediate effect on prosthesis control. 
These devices are hermetically encapsulated, lead- 
less electrical devices that are small enough to be 
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Figure 19.2. Current state-of-the-art prosthesis for a man with bilateral shoulder 
disarticulation amputations. 



injected percutaneously into muscles (2 mm diam- 
eter x 15 mm long). They receive their power, digi- 
tal addressing, and command signals from an 
external transmitter coil worn by the patient. The 
hermetically sealed capsule and electrodes neces- 
sary for long-term survival in the body have re- 
ceived investigational device approval from the 
FDA for experimental use in people with func- 
tional electrical stimulation systems. As such, these 
BIONs represent an enabling technology for a 
prosthesis control system based on implanted my- 
oelectric sensors. 

Weir, Troyk, DeMichele, and Kuiken (2003) 
are involved in an effort to revisit (Reilly 1973) the 
idea of implantable myoelectric sensors. Im- 
plantable myoelectric sensors (IMES) have been 
designed that will be implanted into the muscles of 
the forearm and will transcutaneously couple via a 
magnetic link to an external exciter/r data teleme- 
try reader. Each IMES will be packaged in a BION 
II (Arcos et al., 2002) hermetic ceramic capsule 



(figure 19.3). The external exciter/data telemetry 
reader consists of an antenna coil laminated into a 
prosthetic interface so that the coil encircles the 
IMES. No percutaneous wires will cross the skin. 
The prosthesis controller will take the output of an 
exciter/data telemetry reader and use this output to 
decipher user intent. While it is possible to locate 
three, possibly four, independent (free of cross 
talk) surface electromyographic (EMG) sites on the 
residual limb, it will be feasible to create many 
more independent EMG sites in the same residual 
limb using implanted sensors. There are 18 mus- 
cles in the forearm that are involved in the control 
of the hand and wrist. Using intramuscular signals 
in this manner means the muscles are effectively 
acting as biological amplifiers for the descending 
neural commands (figure 19.4). Neural signals are 
on the order of microvolts while muscle or EMG 
signals are on the order of millivolts. Intramuscu- 
lar EMG signals from multiple residual muscles of- 
fer a means of providing simultaneous control of 



298 



Technology Applications 



TRANSMITTER 



TELEMETRY 
CONTROLLER 




PROSTHETIC 
HAND 



PROSTHESIS 
CONTROLLER 



RESIDUAL 
LIMB 



PROSTHETIC 

INTERFACE 

(SOCKET) 



Figure 19.3. Schematic of how the implantable myoelectric sensors will be located within the fore- 
arm and encircled by the telemetry coil when the prosthesis is donned. 



multiple degrees of freedom in a multifunction 
prosthetic hand. 

At levels of amputation above the elbow, the 
work of Kuiken (2003) offers a promising surgical 
technique to create physiologically appropriate 
control sites. He advocates the use of "targeted 
muscle reinnervation" or neuromuscular reinner- 
vation to improve the control of artificial arms. 
Kuiken observed that although the limb is lost in 
an amputation, the control signals to the limb re- 
main in the residual peripheral nerves of the am- 
putated limb. 

The potential exists to tap into these lost con- 
trol signals using nerve-muscle grafts. As first sug- 



gested by Hoffer and Loeb (1980), it is possible to 
denervate expendable regions of muscle in or near 
an amputated limb and graft the residual periph- 
eral nerve stumps to these muscles. The peripheral 
nerves then reinnervate the muscles and these 
nerve-muscle grafts would provide additional EMG 
control sites for an externally powered prosthesis. 
Furthermore, these signals relate directly to the 
original function of the limb. 

In the case of the high-level amputee, the me- 
dian, ulnar, radial, and musculocutaneous nerves 
are usually still present (figure 19.5). The musculo- 
cutaneous nerve controls elbow flexion, while the 
radial nerve controls extension. Pronation of the 



Figure 19.4. A muscle as a biological 
amplifier of the descending neural 
command. CNS, central nervous 
system. 
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Figure 1 9.5. Schematic of neuromuscular reinnervation showing how the pec- 
toralis muscle is split into four separate sections and reinnervated with the 
four main arm nerve bundles. See also color insert. 



forearm is directed by the median nerve and 
supination by the radial and musculocutaneous 
nerve. Extension of the hand is governed by the ra- 
dial nerve and flexion by the median and ulnar 
nerves. Since each of these nerves innervates mus- 
cles that control the motion of different degrees of 
freedom, they should theoretically supply at least 
four independent control signals. The nerves are 
controlling functions in the prosthesis that are di- 
rectly related to their normal anatomical function. 

IMES located at the time of initial surgery 
would complement this procedure nicely by pro- 
viding focal recording of sites that may be located 
physically close together. Targeted muscle reinner- 
vation has been successfully applied in two tran- 
shumeral amputees and one bilateral, high-level 
subject to control a shoulder disarticulation pros- 
thesis. In the shoulder disarticulation case, each of 
the residual brachial plexus nerves wsa grafted to 
different regions of denervated pectoralis muscle, 
and in the transhumeral cases the nerves were 
grafted to different regions of denervated biceps 
brachii. 

Kuiken's neuromuscular reorganization tech- 
nique is being performed experimentally in people. 
IMES are due to be soon available for human test- 
ing. However, the problem is not solved. Now that 
multiple control signals are available for control, 
we need to figure out how to use the information 



they provide to control a multifunction prosthesis 
in a meaningful way. 



The CNS as a Source of Control Signals: 
The Brain-Computer Interface 

The area of brain-computer interfaces (BCI) has at- 
tracted considerable attention in the last few years, 
with various demonstrations of brain-derived sig- 
nals used to control external devices in both ani- 
mals and humans. Potential applications include 
communications systems for "locked-in" patients 
suffering from complete paralysis due to brain stem 
stroke or ALS, and environmental controls, com- 
puter cursor controls, assistive robotics, and FES 
systems for spinal cord injury patients (see chap- 
ters 20 and 22, this volume). The field encom- 
passes invasive approaches that require the surgical 
implantation of recording electrodes either beneath 
the skull or actually within the cortex, and nonin- 
vasive approaches based on potentials recorded 
over the scalp (figure 19.6). The latter approach 
has much less spatial and temporal resolution and 
consequently less potential bandwidth than the 
signals available from intracortical or even epidural 
electrodes. As a result, there is a trade-off between 
the quality of the signals and the risk to which a 
patient would be exposed. Issues of convenience 
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Figure 19.6. Noninvasive and invasive brain-computer interfaces. (A) Electroencephalographic (EEG) signals 
recorded from the scalp have been used to provide communication or other environmental controls to locked-in 
patients, who are completely paralyzed due to brain stem stroke or neurodegenerative disease. The signals are am- 
plified and processed by a computer such that the patient can learn to control the position of a cursor on a screen 
in one or two dimensions. Among other options, this technique can be used to select letters from a menu in order 
to spell words. From Kubler et al. (2001). (B) Monkeys have learned to control the 3-D location of a cursor (yellow 
sphere) in a virtual environment. The cursor and fixed targets are projected onto a mirror in front of the monkey. 
The cursor position can be controlled either by movements of the monkey's hand or by the hand movement pre- 
dicted in real time on the basis of neuronal discharge recorded from electrodes implanted in the cerebral cortex. 
From Taylor et al. (2002). See also color insert. 



and patient acceptability also come into play. Ex- 
ternal recording electrodes can be unsightly and 
uncomfortable and require much caregiver assis- 
tance to don and maintain. The state of the field is 
that a range of useful technologies are being devel- 
oped to accommodate the priorities and concerns 
of a varied group of potential users. 

EEG Recordings 

For more than 20 years, the most systematic at- 
tempts at clinical application of BCIs to the sensory- 
motor system have used specific components of 
EEG signals (Niedermeyer & Lopes Da Sylva, 1998). 
A few BCI systems use involuntary neural responses 
to stimuli to provide some rudimentary access 
to the thoughts and wishes of the most severely 
paralyzed individuals (Donchin, Spencer, & Wi- 
jesinghe, 2000). However, for paralyzed individuals 
with a relatively intact motor cortex, there is evi- 
dence that volitional command signals can be pro- 
duced in association with attempted or imagined 
movements. Characteristic changes in the EEG in 
the u (8-12 Hz) and (3 (18-26 Hz) bands usually 



accompany movement preparation, execution, and 
termination (Babiloni et al, 1999; Pregenzer & 
Pfurtscheller, 1999; Wolpaw, Birbaumer, McFarland, 
Pfurtscheller, & Vaughan, 2002) and can still be 
observed years after injury (Lacourse, Cohen, 
Lawrence, & Romero, 1999). However, fine details 
about the intended movement, such as direction, 
speed, limb configuration, and so on, are much 
more difficult to discern from EEG than from intra- 
cortical unit recordings. 

There are two primary ways these sensorimotor 
rhythms have been utilized in BCIs. One option is 
to use discrete classifiers that recognize spatiotem- 
poral patterns associated with specific attempted 
movements (Birch, Mason, & Borisoff, 2003; 
Blankertz et al., 2003; Cincotti et al., 2003; Garrett, 
Peterson, Anderson, & Thaut, 2003; Graimann, 
Huggins, Levine, & Pfurtscheller, 2004; Millan & 
Mourino, 2003; Obermaier, Guger, & Pfurtscheller, 
1999). This approach might allow selection be- 
tween several different options or states of a device, 
but it cannot provide continuous, proportional 
control. The second option is to translate sensori- 
motor rhythms continuously into one or more 
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proportional command signals. In 1994, Wolpaw et 
al. demonstrated x- and y-axis cursor control using 
the sum and difference respectively, of the |X-range 
power recorded from left and right sensorimotor ar- 
eas (Wolpaw & McFarland, 1994). More recently, 
this group has improved this paradigm by adap- 
tively refining the EEG decoder during the training 
process. The decoder's coefficients were regularly 
adjusted in order to make use of learning-induced 
changes in the person's EEG modulation capabili- 
ties (Wolpaw & McFarland, 2004). In that study, 
four subjects, including two spinal injury patients, 
learned to make fairly accurate 2-D cursor move- 
ments. This result with injured patients is particu- 
larly important, as it demonstrates that the ability to 
modulate activity in primary motor cortex (Ml) 
voluntarily apparently survives spinal cord injury. 
Additional evidence from both EEG and func- 
tional magnetic resonance imaging indicates that 
the motor cortex can still be activated by imagined 
movements even years after a spinal cord injury 
(Lacourse et al., 1999; Shoham, Halgren, Maynard, 
& Normann, 2001). 

Intracortical Recordings 

While EEG signals show increasing promise, it 
seems reasonable to anticipate that the more inva- 
sive intracortical recordings offer more degrees of 
freedom and more natural control than does EEG. 
This conclusion is supported by the commonsense 
observation that EEG signals are heavily filtered 
products of cortical activities. However, a direct 
comparison of the two approaches has yet to be 
made. The information transmission rate calcu- 
lated in a limited number of separate experiments 
allows an indirect comparison to be made. A rate of 
0.5 bits per second (bps) has been achieved from 
EEG recordings (Blankertz et al., 2003), and 
1.5 bps through intracortical recordings (Taylor, 
Tillery, & Schwartz, 2003). 

Understanding the relation between time- 
varying neuronal discharge of single neurons and 
voluntary movement has been a mainstay of motor 
systems studies since the pioneering work of Evarts 
(1966). Within 4 years of that research, Humphrey, 
Schmidt, and Thompson (1970) showed that lin- 
ear combinations of three to five simultaneously 
recorded cells could significantly improve the esti- 
mates afforded by single-neuron recordings. About 
the same time, in one of the earliest examples of in- 



tracortical control, monkeys learned to control the 
firing rate of individual neurons in Ml on the basis 
of simple visual feedback (Fetz, 1969). A number 
of years later, a simple two- or three-electrode sys- 
tem intended to provide rudimentary communica- 
tion was implanted in several locked-in human 
patients (Kennedy & Bakay, 1998). 

Within the past 5 years, technologies for mi- 
croelectrode recording and signal processing have 
developed to the point that it is now feasible to use 
simultaneous recordings from nearly one or two 
orders of magnitude more cells. High-density ar- 
rays of electrodes implanted in Ml or premotor ar- 
eas have been used to control both virtual and real 
robotic devices. In the first such system, Chapin 
and coworkers trained rats to retrieve drops of wa- 
ter by pressing a lever controlling the rotation of a 
robotic arm (Chapin, Moxon, Markowitz, & 
Nicolelis, 1999). They used the activities of 21 to 
46 neurons, recorded with microwires implanted 
in Ml, as input to a computer program which con- 
trolled the motion of the robot. Several rats learned 
to operate the arm using the neural signals, with- 
out actually moving their own limbs. 

More recently, monkeys have controlled multi- 
joint physical robotic arms (Carmena et al, 2003; 
Taylor et al., 2003; Wessberg et al., 2000) and vir- 
tual devices in two and three dimensions (Musal- 
lam, Corneil, Greger, Scherberger, & Andersen, 
2004; Serruya, Hatsopoulos, Paninski, Fellows, & 
Donoghue, 2002; Taylor, Tillery, & Schwartz, 
2002). In these experiments, as many as 100 elec- 
trodes were implanted into the cerebral cortex and 
control was based on activities of 10 to 100 neu- 
rons. In early 2004, Cyberkinetics Inc. received 
FDA approval to implant chronic intracortical mi- 
croelectrode arrays in human patients with high- 
level spinal cord injuries. The first patient was 
implanted several months later, several years after 
the original injury. After several months of practice 
and testing, the patient achieved sufficient control 
to play a very simple video game using the inter- 
face (Serruya, Caplan, Saleh, Morris, & Donoghue, 
2004). 

As with EEG signals, it is possible to use intra- 
cortical recordings for either classification or contin- 
uous control. Andersen and coworkers (Shenoy et 
al, 2003) implanted electrodes in the posterior pari- 
etal cortex, a region that is believed to participate in 
movement planning. Monkeys were required to 
reach toward one of two targets displayed on a 
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Figure 19.7. Reconstruction of electromyographic (EMG) activity based on motor cortical sig- 
nals. The blue traces are EMG signals recorded from four arm muscles of a monkey during the 
execution of a multitarget button-pressing task. The green traces were obtained from neural sig- 
nals recorded in the primary motor cortex (Ml). The relationship between recorded Ml activity 
and EMGs was estimated using multiple input least-squares system identification techniques, 
resulting in a set of optimal nonparametric linear filters. Principal component analysis was first 
used to reduce the complexity of the inputs to these models. To make the filters more robust, a 
singular value decomposition pseudo-inverse of the input autocorrelation was employed. The 
resulting filters were then used to predict EMG activity in separate data sets to test the stability 
and generalization of the correlation between Ml and EMG. 



touch screen, and a probabilistic algorithm pre- 
dicted the preferred target based on discharge 
recorded during the delay period preceding move- 
ment. Within 50 trials, the monkeys learned to 
modulate the discharge in order to indicate the in- 
tended target in the absence of any limb movement. 
Notably, however, none of the BCI-based con- 
trol systems implemented to date has reflected the 
dynamics of the arm. Instead they have used a posi- 
tion or velocity controller to drive a reference point 
whose location is instantaneously derived from 
neuronal activity. Yet there is abundant evidence 
that the activity of most Ml neurons is affected by 
forces exerted at the hand (Ashe, 1997; Evarts, 1969; 
Kalaska, Cohon, Hyde, & Prud'homme, 1989) and 
by the posture of the limb (Caminiti, Johnson, & 
Urbano, 1990; Kakei, Hoffman, & Strick, 1999; 
Scott & Kalaska, 1995). One report did describe 
the control of a single-degree-of-freedom force 



gripper though Ml recordings from a rhesus mon- 
key (Carmena et al., 2003). In those experiments, 
Ml discharge typically accounted for more of the 
variance within the force signal than it did for either 
hand position or velocity. 

Beyond the prediction of a single-degree-of- 
freedom force signal, we have shown that rectified 
and filtered EMG signals can be predicted on the ba- 
sis of 40 single- and multiunit intracortical record- 
ings (Pohlmeyer, Miller, Mussa-Ivaldi, Perreault, & 
Solla, 2003). In those experiments, the activity of as 
many as four muscles of the arm and hand was pre- 
dicted simultaneously, accounting for between 60% 
and 75% of the total variance during a series of but- 
ton presses. A typical example of the correspon- 
dence between actual and predicted EMG is shown 
in figure 19.7. In another series of very recent exper- 
iments, we estimated the torques generated during a 
series of planar, random-walk movements. We 
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calculated predictions of Cartesian hand position, as 
well as shoulder and elbow joint torque using the 
discharge of 88 single neurons recorded from Ml 
and dorsal premotor cortex (PMd). On average, 
these data accounted for 78% of the torque variance 
but only 53% of the hand position variance. The im- 
plications for BCI control are significant. If the mo- 
tor commands expressed by Ml neurons have a 
substantial kinetic component, using them in a sys- 
tem that emulates the dynamics of the limb might 
provide more natural control. 



Feedback Is Needed for Learning 
and for Control 

Not surprisingly, the success of the experiments de- 
scribed above was substantially improved by real- 
time feedback of performance. Feedback normally 
allows for two corrective mechanisms. One is the 
online correction of errors; the other is the gradual 
adaptation of motor commands across trials. The 
latter mechanism has been extensively studied in 
both humans and monkeys (Scheidt, Dingwell, & 
Mussa-Ivaldi, 2001; Shadmehr & Mussa-Ivaldi, 
1994; Thoroughman & Shadmehr, 2000). 

The long delays intrinsic to the visual system 
(100-200 milliseconds) make it unsuitable for the 
online correction of errors in a complex dynamic 
system like the human arm, except during move- 
ments that are much slower than normal. This 
problem has been partially sidestepped in most 
brain-machine interface experiments by eliminat- 
ing plant dynamics — for example, by controlling a 
cursor or virtual limb on a computer monitor or by 
means of a servomechanism enforcing a com- 
manded trajectory. While these approaches elimi- 
nate mechanical dynamics, they do not eliminate 
the neural dynamics responsible for the generation 
of the open-loop signals. As a consequence, com- 
putational errors can only be corrected after long 
latencies. This is likely to be one of the causes of 
tracking inaccuracy. If the muscles of a paralyzed 
patient were to be activated through a BCI, the dy- 
namics of the musculoskeletal system would need 
to be considered. 

While visual feedback plays an important role 
in the planning and adaptation of movement, other 
sensory systems ordinarily supply more timely in- 
formation. Recordings from peripheral sensory 
nerves have been used as a source of feedback for a 



closed-loop FES system for controlling hand grip 
in a patient (Inmann, Haugland, Haase, Biering- 
Sorensen, & Sinkjaer, 2001). Signals derived from 
a controlled robot might instead be used to stimu- 
late these nerves as a means of approximating 
the natural feedback during reaching. However, in 
most of the situations in which such systems 
would be clinically useful, conduction through 
these nerves to the central nervous system is not 
present. The auditory system might provide an- 
other rapid pathway into the brain. Much effort has 
also been expended in the development of a visual 
prosthesis, including attempts to stimulate both 
the visual cortex (Bak et al., 1990; Normann, May- 
nard, Rousche, & Warren, 1999) and the retina 
(Zrenner, 2002). The latter methods, while in sev- 
eral ways more promising for blind patients, are 
unlikely to provide useful feedback for a motor 
prosthesis, as they would suffer nearly as long a la- 
tency as the normal visual system. Direct cortical 
stimulation might substantially decrease the delay, 
but mimicking the sophisticated visual signal pro- 
cessing of the peripheral nervous system is a 
daunting prospect. 

The somatosensory system, including propri- 
oception, offers a more natural modality for 
movement-related feedback. Limited experimental 
efforts have been made to investigate the percep- 
tual effects of electrical stimulation in the so- 
matosensory cortex. Monkeys have proven capable 
of distinguishing different frequencies of stimula- 
tion, whether applied mechanically to the fingertip 
or electrically to the cortex (Romo, Hernandez, 
Zainos, Brody, & Lemus, 2000). In one demonstra- 
tion, the temporal association of electrical stimuli 
to somatosensory cortex (cue) with stimuli to the 
medial forebrain bundle (reward) conditioned 
freely roaming rats to execute remotely controlled 
turns (Talwar et al., 2002). It is not yet known 
whether cortical stimulation could provide ade- 
quate feedback to guide movement in the absence 
of normal proprioception. 



The Clinical Impact of 
Neural Engineering 

Cochlear Implants 

As outlined earlier in this chapter, the cochlear im- 
plant is perhaps the best modern-day example of a 



304 



Technology Applications 



successful application of neural engineering design 
and technology that has clearly generated strong 
benefits for its recipients. The impact of the cochlear 
implant has been revolutionary in that a relatively 
simple design has generated substantial enhance- 
ment in hearing for its recipients. Predictably, this 
device has also raised expectations for other clini- 
cal applications of neural engineering; however, 
these have not yet been fulfilled. Nonetheless, our 
experience with cochlear implants is salutary in 
many respects. 

First, our experience has verified that im- 
plantable neural stimulators could function effec- 
tively for long periods of time. Although implanted 
battery-powered stimulators have been available 
for a variety of applications for many years (most 
notably for cardiac pacing in complete heart block 
or other cardiac arrhythmias), the cochlear implant 
is the first fully implantable neural system shown 
to operate effectively without failure or significant 
disruption of function. 

Second, the sophistication of the electrode de- 
signs, placement, and stimulus protocols that were 
utilized in the early stages of this technology were 
quite primitive, especially when compared with 
the elegant design and intrinsic function of the 
natural biological organ (i.e., the cochlea) itself. 
Nonetheless, the clinical success of the implant il- 
lustrated that relatively unsophisticated technolo- 
gies could still prove to be very useful for treating 
both congenital and acquired hearing loss. From 
this, we can deduce that neural plasticity and 
adaptation appear to be extremely important in 
generating this success, and that the inherent neu- 
ral plasticity present in most nervous system ele- 
ments is extremely important and can readily 
compensate for considerable technical or design 
inadequacies. 

The third lesson is that the devices have proved 
to be relatively inexpensive and readily accepted by 
clinicians and consumers, providing a framework 
for successful design and implementation of other 
neural engineering devices in the future. 

EEG Based BCIs 

EEG-based BCIs have been a focus of intensive re- 
search for several decades (see also chapter 20, this 
volume). In the beginning, relatively simple mea- 
sures of EEG root-mean squared (RMS) amplitude 



or of spectral content were utilized to drive a cursor 
on a computer screen or to generate binary choices 
for controlling electrical or mechanical systems or 
devices. The signal processing time was often quite 
long and the error rates high, so that many patients 
were reluctant to utilize these approaches because 
these factors caused considerable frustration. 

To optimize the utility of these approaches, a 
number of time-efficient and elegant signal pro- 
cessing and classification algorithms have been ad- 
vanced, and many appear to be able to accelerate 
the speed with which binary or ternary choices can 
be made. There is, however, no well-documented 
example in which surface EEG signals have been 
shown to have appropriate input-output properties 
in that signal power or some other measured mag- 
nitude component could be linked to the ampli- 
tude or speed of the desired outcome measure 
(such as speed of motion or muscle force). Such 
improvements are likely to be necessary if natural 
and spontaneous scaling of motion is to occur. 

In spite of these difficulties, certain impair- 
ments such as high quadriplegia or widespread 
neural or muscular loss with amyotrophic lateral 
sclerosis or Guillian-Barre syndrome may require 
large-scale substitution of neuromuscular function. 
Under these conditions, the time delays that are 
characteristic of many BCI systems are potentially 
less onerous, and many patients appear willing to 
deal with these inadequacies because they have few 
acceptable alternatives. 

A related but somewhat different approach is 
to use EEG signals to control implanted functional 
neuromuscular stimulation systems, either for 
upper-extremity hand control or for lower-extremity 
function in standing up. When real-time control 
approaches are attempted, the time constraints and 
scaling nonlinearities become more pressing, and 
there is as yet no widespread application of these 
techniques. 

Implanted EEG Recording Systems 

There is an understandable reluctance to insert 
electrodes chronically into undamaged human ce- 
rebral cortical tissues because of the potential for 
bleeding, infection, and motion-related damage of 
cortical neurons that could lead to scarring. As an 
alternative, recording through transcalvarial EEG 
electrodes has considerable appeal, in that the dura 
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remains intact, yet signal amplitude rises substan- 
tially, making efficient signal processing tasks more 
feasible. 

Implanted Unitary Recording Systems 

As outlined earlier, promising examples of im- 
plantable human cortical electrode systems are also 
now emerging that focus either on unitary record- 
ings from the motor cortex (Serruya et al., 2004) or 
on multiunit recordings and field responses from 
parietal areas of the cortex (Musallam et al, 2004; 
Shenoy et al, 2003). Although these approaches 
are promising in that they may allow the physician 
to tap the sources of the command signals directly, 
this work is still in its infancy, and we know little 
about the benefits and potential complications of 
such approaches. 

Implanted Nerve Cuffs 

Other long-standing neural engineering approaches 
that have promise include implantation of nerve 
cuffs around cutaneous nerves such as the sural 
nerve to detect foot contact during gait, a useful 
measure for controlling functional electrical stimu- 
lation. Additional novel peripheral recording sys- 
tems, described earlier, include the use of EMG 
recordings from muscles that have received im- 
planted peripheral nerves after an amputation 
(Kuiken, 2003). The muscles receiving such nerves 
act as biological amplifiers, providing useful signals 
for prosthesis control. 

Implanted Stimulation Systems 

It is also clear that electrical stimulation may be 
combined with intensive physical training to aug- 
ment improvements in performance such as arise 
in Nudo's work on stroke recovery in animal mod- 
els and in parallel studies in human brain stimula- 
tion using the Northstar Neuroscience system 
(Nudo, Plautz, & Frost, 2001; Nudo, Wise, Si- 
fuentes, & Milliken, 1996; Plautz et al, 2002). 

It has become evident that significant improve- 
ment in performance may rely on remapping and 
reorganization of cortical tissues located in the 
neighborhood of the stroke and that these cortical 
systems can be retrained optimally using a variety 
of interventional methods. 



The Brain-Computer Metaphor: 
Biomimetic and Hybrid Technologies 

During the past century, studies of computers and 
of the brain have evolved in a reciprocal metaphor: 
The brain is seen as an organ that processes infor- 
mation and computers are developed in imitation 
of the brain. Despite the speed with which today's 
computers execute billions of operations, their bio- 
logical counterparts have unsurpassed performance 
when it comes to recognizing a familiar face or con- 
trolling the complex dynamics of the arm. Hence, 
the computational power of biological systems has 
sparked intense activity aimed at mimicking neu- 
robiological processes in artificial systems. 

Two distinguishing features of biological sys- 
tems are guiding the development of biomimetic 
research: (a) their ability to adapt to novel condi- 
tions without the need for being "reprogrammed" 
by some external agent, and (b) their ability to re- 
generate and reproduce themselves. In addition, 
biological systems use massively parallel computa- 
tion systems. The first feature is particularly rele- 
vant to the creation of machines that can explore 
remote and dangerous environments, such as the 
surface of other planets or deep undersea environ- 
ments. 

The ability to reproduce and regenerate is per- 
haps the most visible defining feature of living or- 
ganisms. And indeed, while one may have the 
impression that materials like metals and plastics 
are superior to flesh and bones, there is hardly any 
human artifact that can continuously operate for 
several decades without irreparably breaking down. 
The durability of biological tissue is ensured by the 
complex molecular mechanisms that underlie their 
continuous renewal. And the biological mechanisms 
of reproduction are an object of growing interest 
in robotics, where the theme of self-assembling 
devices is making promising strides (Murata, 
Kurokawa, & Kokaji, 1994). 

While mimicking the nervous system has led 
to the creation of artificial neural networks, a dif- 
ferent idea has begun to take shape: to construct 
hybrid computers in which neurons are grown 
over a semiconductor substrate (Fromherz, Offen- 
hausser, Vetter, & J., 1991; Fusi, Annunziato, 
Badoni, Salamon, & Amit, 2000; Grattarola & Mar- 
tinoia, 1993; Zeck & Fromherz, 2001). Fromhertz's 
team has developed a simple prototype, in which 
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electrical signals are delivered by the substrate to a 
nerve cell. The responses are transmitted via an 
electrical synapse to a second cell, and the activity 
of the second cell is read out by the semiconductor 
substrate. Chiappalone et al. (2003) have grown 
cell cultures from chick embryos over microelec- 
trode arrays and have succeeded inducing plastic 
changes by applying drugs that acted on glutamate 
receptors. 



Neural Plasticity as 

a Programming Language 

A neurobiological basis for programming brain- 
machine interactions is offered by the different 
mechanisms of neural plasticity, such as long-term 
potentiation (LTP; Bliss & Collingridge, 1993), 
long-term depression (LTD; Ito, 1989), short-term 
memory (Zucker, 1989), and habituation (Castel- 
lucci, Pinsker, Kupfermann, & Kandel, 1970). A 
common feature of the different forms of plasticity 
is their history-dependence: In addition to what is 
known as phyletic memory (Fuster, 1995; i.e., in- 
structions "written" in the genetic code), neural 
systems are programmed by their individual expe- 
rience. A clear example of this is offered by Heb- 
bian mechanisms that relate the strengthening or 
weakening of synaptic connections to the temporal 
correlation of pre- and postsynaptic activations: If 
the firing of a presynaptic neuron is followed by 
firing of the postsynaptic neuron, then the efficacy 
of the synapse increases (LTP). Conversely, if the 
firing of the postsynaptic neuron occurs when the 
presynaptic neuron is silent, the connection efficacy 
is decreased (LTD). From an operational stand- 
point, these forms of plasticity can be regarded as 
an assembly language for the neural component in 
a brain-machine interaction. 

To gain access to this "programming language," 
it is not sufficient to observe that some form of plas- 
ticity must take place, for example, in the operant 
conditioning of motor-cortical signals (Fetz, 1969; 
Wessberg et al., 2000). A challenge that has yet to 
be met is to acquire the means for obtaining the de- 
sired efficacy of the synaptic connections between 
neurons and neuronal populations at specific loca- 
tions of the nervous system. This would correspond 
to acquiring the ability to design the behavior of bi- 
ological neural network, as we know how to design 
the behavior of artificial neural networks. 



Brain-Machine Interactions for 
Understanding Neural Computation 

Some investigators have used the closed-loop inter- 
action between nerve cells and external devices as a 
means to study neural information processing 
(Kositsky Karniel, Alford, Fleming, & Mussa-Ivaldi, 
2003; Potter, 2001; Reger, Fleming, Sanguineti, Al- 
ford, & Mussa-Ivaldi, 2000; Zelenin, Deliagina, 
Grillner, & Orlovsky, 2000). Mussa-Ivaldi and 
coworkers (Reger et al., 2000) have established 
bidirectional connections between a robotic device 
and a lampreys brain stem (figure 19.8) with the 
goal of understanding the operations carried out by 
a single layer of connections in the reticular forma- 
tion. Signals generated by the left and right optical 
sensors of a small mobile robot were translated into 
electrical stimuli with frequency proportional to the 
light intensity. The stimuli were applied to the path- 
ways connecting the lampreys right and left vestibu- 
lar organs to two populations of reticular neurons. A 
simple interface translated the resultant discharge 
frequency of the reticular neurons into motor com- 
mands to the robots right and left wheels. In this 
simple arrangement, the reticular neurons acted as a 
processing element that determined the closed-loop 
response of the neuro-robotic system to a source of 
light. These studies revealed that (a) different be- 
havior can be generated by different electrode loca- 
tions; (b) the input-output relation of the reticular 
synapses are well approximated by simple linear 
models with a recurrent dynamical component; and 
(c) the prolonged suppression of one input channel 
leads to altered responsiveness well after it has been 
restored. 

In a similar experiment, Deliagina, Grillner, and 
coworkers (Zelenin et al., 2000) used the activity 
of reticulo-spinal neurons recorded from a swim- 
ming lamprey to rotate the platform supporting the 
fish tank. The lamprey was able to stabilize the 
hybrid system, and this compensatory effect was 
most efficient in combination with undulatory 
swimming motions. These studies demonstrate the 
feasibility of closed-loop interactions between a 
specific region of the nervous system and an artifi- 
cial device. Closed-loop BCIs offer the unparalleled 
possibility to replace the neural system with a com- 
putational model having the same input-output 
structure, thus providing a direct means for testing 
the predictions of specific hypotheses about neural 
information processing. 
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Figure 19.8. A hybrid neurorobotic system. Signals from the optical sensors of 
Khepera (K-team) mobile robot (bottom) are encoded by the interface into electrical 
stimulations whose frequency depends linearly upon the light intensity. These stimuli 
are delivered by tungsten microelectrodes to the right and left vestibular pathways of 
a lamprey's brain stem (top) immersed in artificial cerebrospinal fluid within a 
recording chamber. Glass microelectrodes record extracellular responses to the stim- 
uli from the posterior rhomboencephalic neurons (PRRN). Recorded signals from 
right and left PRRNs are decoded by the interface, which generates the commands to 
the robot's wheels. These commands are set to be proportional to the estimated aver- 
age firing rate on the corresponding side of the lamprey's brain stem. See also color 
insert. 



Conclusion 

It is self-evident that neural engineering is a multi- 
disciplinary endeavor. Perhaps what makes it even 
more so is the fact that both components of it — 
neuroscience and engineering — arise in their own 
right from a combination of disciplines, such as 



molecular biology, electrophysiology, mathematics, 
signal processing, physics, and psychology. In this 
chapter, we have attempted to paint a portrait of 
neural engineering based on its dual character. 

On one hand, we are seeing a rapid develop- 
ment of approaches that involve the use of brain- 
machine interaction to improve the living conditions 



308 



Technology Applications 



of severely disabled patients. Remarkably, this type 
of intervention is not merely aimed at restoring 
functions. Brain-machine and brain-computer in- 
terfaces can actually open the door to functions 
that are not naturally available. For example, neu- 
ral signals can be used in combination with 
telecommunication technologies for controlling re- 
mote devices or for transmitting (and receiving) in- 
formation across geographical distances. This will 
impact not only patients' lives but also the activi- 
ties of health care providers. Surgical procedures 
have been already carried out via radio link con- 
necting physician and patient across the Atlantic 
Ocean (Marescaux et al., 2001). We may expect 
that the possibility of monitoring the actions of a 
surgeon to a high level of detail — eye movements, 
hand movements, hand forces, and so on — will 
bring new advances in telemedicine. 

On the other hand, we are witnessing attempts 
at interfacing with the nervous system with the in- 
tention of capturing its power. As Steve DeWeerth 
puts it, one may consider "neurobiology as a tech- 
nology" and neural engineering as an attempt at ac- 
cessing this technology by creating hybrid systems 
(personal communication). Can we possibly suc- 
ceed? Perhaps this is not the right question. Better 
questions are: To what extent we may succeed, and 
what may failures teach us about both technology 
and biology? 

Failures along this difficult way will certainly 
occur because our knowledge and models of brain 
functions are almost certainly flawed and un- 
questionably limited. For example, on one hand 
computational neuroscientists describe synaptic 
connections using real valued numbers or weight- 
sto describe the efficacy of transmission. But even 
accepting this view, we do not really know what 
ranges of values are possible. Can synaptic connec- 
tions be approximated by continuous variables? 
On the other hand, neurobiologists describe plas- 
ticity as a basis for learning and memory and find 
that postsynaptic potentials may be altered by par- 
ticular patterns of stimuli. But how are these 
changes reflected in the spiking activities of neu- 
rons? And how is it possible to consistently drive a 
cell or a population to a desired level of responsive- 
ness? By exposing the difficulties in achieving con- 
trolled brain-machine interactions and by 
challenging artificial divisions of labor between 
theoretical and experimental work, neural engi- 
neering may become a driving force in neuroer- 



gonomics, neurobiology, and computational neuro- 
science. 



Main Points 

1 . Neural engineering is a rapidly growing 
research field that arises from the marriage of 
engineering and neuroscience. A specific goal 
of neural engineering is to establish direct and 
operational interactions between the nervous 
system and artificial devices. 

2. The area of brain-computer interfaces has 
attracted considerable attention, with various 
demonstrations of brain-derived signals used 
to control external devices in both animals and 
humans. High-density arrays of electrodes 
implanted in the cerebral cortex have been 
used to control both virtual and real robotic 
devices. Potential applications include 
communications systems for patients suffering 
from complete paralysis due to brain stem 
stroke and environmental controls, computer 
cursor controls, and assistive robotics for 
spinal cord injury patients. 

3. Success in dealing with the transmission of 
sensory information to central brain structures 
will have critical implications for the use of 
artificial electrical stimuli as a source of 
feedback to the motor system. Sensory 
feedback allows for two corrective 
mechanisms. One is the online correction of 
errors; the other is the gradual adaptation of 
motor commands across trials. Signals derived 
from a controlled robot might be used to 
stimulate these nerves as a means of 
approximating the natural feedback. 

4. Neural plasticity and adaptation are of critical 
importance for the success of BCIs. The 
inherent neural plasticity present in most ner- 
vous system elements can compensate for 
considerable technical or design inadequacies. 
From an operational standpoint, these forms 
of plasticity can be regarded as an assembly 
language for the neural component in a brain- 
machine interaction. 

5. As we see a rapid development of approaches 
that involve the use of brain-machine 
interactions to improve the living conditions 
of severely disabled patients, this is not the 
only value of BCIs. One may consider 
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neurobiology as a technology and neural 
engineering as an attempt at accessing this 
technology by creating hybrid systems. In this 
endeavor, brain-machine interfaces are a new 
and unparalleled tool for investigating neural 
information processing. 
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EEG Based Brain-Computer 
Interface 



A brain-computer interface (BCI) is a system that 
allows its user to interact with his environment 
without the use of muscular activity as, for exam- 
ple, hand, foot, or mouth movement (Wolpaw et 
al, 2002). This would mean that a specific type of 
mental activity and strategy are necessary to mod- 
ify brain signals in a predictable way These sig- 
nals have to be recorded, analyzed, classified, and 
transformed into a control signal at the output of 
the BCI. From the technical point of view, a BCI 
has to classify brain activity patterns online and in 
real time. This process is highly subject specific 
and requires a number of training or learning 
sessions. 

The current and most important application of 
a BCI is to assist patients who have highly limited 
motor functions, such as completely paralyzed pa- 
tients with amyotrophic lateral sclerosis (ALS) or 
high-level spinal cord injury. In the first case, the 
BCI can help to realize a spelling device to facilitate 
communication (Birbaumer et al., 2000), and in 
the second case the BCI can bypass the damaged 
neuromuscular channels to control a neuropros- 
thesis (Pfurtscheller, Muller, Pfurtscheller, Gerner, 
& Rupp, 2003). Further applications occur in neu- 
rofeedback therapy and the promising field of mul- 
timedia and virtual reality applications. 



Components Defining a BCI 

A BCI system is, in general, defined by the follow- 
ing components: type of signal recording, fea- 
ture of the brain signal used for control, mental 
strategy, mode of operation, and feedback (see 
figure 20.1). 

The brain signal can be measured in the form 
of electrical potentials or as blood oxygen level- 
dependent (BOLD) response using real-time func- 
tional magnetic resonance imaging (fMRI; Weiskopf 
et al., 2003; see also chapter 4, this volume). The 
electrical potential can be obtained by direct 
recording from cortical neurons in the form of local 
field potentials and multiunit activity, electrocor- 
ticogram (ECoG), or electroencephalogram (EEG). 
Intracortical electrode arrays are used for direct 
brain potential recording (Nicolelis et al., 2003), 
while electrode arrays and strips are utilized for 
subdural recording (Levine et al, 2000). The main 
difference between EEG and subdural or intracorti- 
cal recordings is that the former is a noninvasive 
method without any risk (with a poor signal-to- 
noise ratio) and the latter is an invasive method 
(but results in a very good signal-to-noise ratio). 

A variety of features can be extracted from elec- 
trical signals. In the case of intracortical recordings, 
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Figure 20.1 . Components of a brain-computer interface (BCI). 



they include, for example, the firing patterns of 
cortical neurons. In EEG and ECoG recordings, ei- 
ther slow cortical potential shifts, components of 
visual evoked potentials, amplitudes of steady-state 
evoked potentials, or dynamic changes of oscilla- 
tory activity are of importance and have to be ana- 
lyzed and classified (Wolpaw et al., 2000). 

One important mental strategy to operate a 
BCI is motor imagery. Others are focused attention 
or operant conditioning. In the case of operant 
conditioning, the subject has to learn (by feedback) 
to produce, for example, negative or positive slow 
cortical potential shifts (Birbaumer et al., 1999). 
Focused attention on a certain visually presented 
item modifies the P300 component of the visual 
evoked potential (VEP; Donchin, Spencer, & Wi- 
jesinghe, 2000) or enhances the amplitude of the 
steady-state visual evoked potential (SSVEP; Mid- 
dendorf, McMillan, Calhoun, & Jones, 2000). Mo- 
tor imagery changes central mu and beta rhythms 
in a way similar to that observed during execution 
of a real movement (Pfurtscheller & Neuper, 
2001). 

The mode of operation determines when the 
user performs, for example, a mental task, thereby 
intending to transmit a message. In principle, there 
are two distinct modes of operation, the first being 
externally paced or cue-based (computer-driven, 
synchronous BCI) and the second internally paced 
or uncued (user-driven, asynchronous BCI). In the 
case of a synchronous BCI, a fixed, predefined time 
window is used. After a visual or auditory cue 
stimulus, the subject has to produce a specific 
brain pattern within a predefined time window (of 



not more than a few seconds) which is simultane- 
ously analyzed. An asynchronous protocol requires 
the continuous analysis and feature extraction of 
the recorded brain signal because the user acts at 
will. Thus, such an asynchronous BCI is in general 
even more demanding and more complex than a 
BCI operating with a fixed timing scheme. 

Feedback is usually presented in the form of a 
visualization of the classifier output or an auditory 
or tactile signal. It is an integral part of the BCI sys- 
tem since the users observe the intended action 
(e.g., a certain movement of a neuroprosthesis) as 
they produce the required brain responses. 

Focused attention to both a visually presented 
character (letter) in a P300 paradigm and to a flick- 
ering item with evaluation of SSVEPs needs gaze 
control. Patients in a late stage of ALS have lost 
such conscious control of eye muscles, and conse- 
quently P300 or SSVEP are not suited for commu- 
nication. We focus therefore on motor imagery as 
mental strategy because no eye control is necessary. 



Motor Imagery Used as Control 
Strategy for a BCI 

Over the last decade, reports based on fMRI 
demonstrated consistently that primary motor and 
premotor areas are involved not only in the execu- 
tion of limb movement but also in the imagination 
thereof (de Charms et al., 2004; Dechent, Mer- 
boldt, & Frahm, 2004; Lotze et al., 1999; Porro 
et al., 1996). It is difficult, however, to determine the 
dynamics of these activities based on such imaging 
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techniques that rely on physiological phenomena 
which lack the necessary fast time response. In 
contrast, EEG and MEG offer this possibility, 
though at the cost of reduced spatial resolution. 

Two types of changes in the electrical activity 
of the cortex may occur during motor imagery: 
One change is time-locked and phase-locked in the 
form of a slow cortical potential shift (Birbaumer 
et al., 1999); the other is time-locked but not 
phase-locked, showing either a desynchronization 
or synchronization of specific frequency compo- 
nents. The term event-related desynchronization 
(ERD) means that an ongoing signal presenting a 
rhythmic component may undergo a decrease in its 
amount of synchrony, reflected by the disappear- 
ance of the spectral peak. Similarly, the term event- 
related synchronization (ERS) only has meaning if 
the event (e.g., motor imagery) is followed by an 
enhanced rhythmicity or spectral peak that was 
initially not detectable (Pfurtscheller & Lopes da 
Silva, 1999). 

By means of quantification of temporal-spatial 
ERD (amplitude decrease) and ERS (amplitude in- 
crease) patterns, it can be shown that motor im- 
agery can induce different types of activation 
patterns, as, for example: 

1 . Desynchronization (ERD) of sensorimotor 
rhythms (mu rhythm and central beta oscilla- 
tions; Pfurtscheller & Neuper, 1997) 

2. Synchronization (ERS) of the mu rhythm 
(Neuper & Pfurtscheller, 2001) 

3. Short-lasting synchronization (ERS) of central 
beta oscillations after termination of motor im- 
agery (Pfurtscheller, Neuper, Brunner, & 
Lopes da Silva, 2005) 

Desynchronization and Synchronization 
of Sensorimotor Rhythms 

It is important to note that motor imagery can mod- 
ify sensorimotor rhythms in a way very similar to 
that observed during the preparatory phase of actual 
movement (Neuper & Pfurtscheller, 1999). This 
means, for example, that imagination of one-sided 
hand or finger movement results in a mu ERD local- 
ized at the contralateral hand representation area. 
Because motor imagery results in somatotropically 
organized activation patterns, mental imagination of 
different movements (e.g., hand, foot, tongue) rep- 
resent an efficient strategy to operate a BCI. 



There is, however, an important point to 
consider. When a naive subject starts to practice 
hand motor imagery, generally only a desynchro- 
nization pattern is found. Desynchronization of 
brain rhythms is a relatively unspecific phenome- 
non and characteristic of most cognitive tasks 
(Klimesch, 1999). This is extremely detrimental for 
a BCI operating in an uncued or asynchronous 
mode, because many false positive decisions will 
result during resting or idling periods. Training 
sessions in which the subject receives feedback 
about the performed mental task are therefore very 
important in BCI research. Furthermore, in the 
case of a simple 2-class motor imagery task with 
imagination of right- versus left-hand movement, 
an ipsilateral localized ERS often develops as the 
number of training sessions increases (Pfurtscheller 
& Neuper, 1997). This is in addition to the usual 
contralateral localized ERD. Generally, such a con- 
tralateral ERD or ipsilateral ERS pattern is associ- 
ated with an increase in the classification accuracy 
of single EEG trials. The example in figure 20.2 
displays the comparison of band power (11-13 Hz) 
time courses (ERD/ERS curves) at two electrode 
positions (left sensorimotor hand area C3 and right 
sensorimotor hand area C4) between an initial ses- 
sion without feedback and a session with feedback. 
It can be clearly seen that initially one-sided hand 
motor imagery revealed only ERD patterns with a 
clear dominance over the contralateral hemisphere. 
After feedback training, however, an ipsilateral ERS 
became apparent. The classification accuracy 
achieved in the training without feedback session, 
computed by means of Fisher's linear discriminant 
analysis (LDA), was 87%. After feedback training, 
the brain patterns could be classified with 100% 
accuracy. 

This example documents, first, the plasticity of 
the brain and the dynamics of brain oscillations 
and, second, the importance of induced brain os- 
cillations or ERS for a high classification accuracy 
in a BCI. 

This interesting observation of a contralateral 
ERD together with an ipsilateral ERS during hand 
motor imagery is the manifestation of a phenome- 
non known as focal ERD/surround ERS (Suffczyn- 
ski, Kalitzin, Pfurtscheller, & Lopes da Silva, 
2001). While the ERD can be seen as a correlate of 
an activated cortical network, the ERS in the upper 
alpha band, at least under certain circumstances, 
can be interpreted as a correlate of a deactivated or 
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Figure 20.2. Band power (11-13 Hz) time courses ±95% confidence interval displaying event-related desynchro- 
nization (ERD) and event-related synchronization (ERS) from training session without feedback (left) and session 
with feedback (right). Data from one able-bodied subject during imagination of left- and right-hand movement. 
Gray areas indicate the time of cue presentation. 



even inhibited network in an anatomically distinct 
area. This interaction between different cortical ar- 
eas can be found not only within the same modal- 
ity, but also between different modalities 
(Pfurtscheller & Lopes da Silva, 1999). The combi- 
nation of the focal ERD and the surround ERS may 
form a suitable neural mechanism for increasing 
neural efficiency, thereby optimizing mental strate- 
gies for BCI control. 

Another example of an intramodal focal 
ERD/surround ERS is found with the imagination of 
foot movement. In this case, neural structures are 
activated in the foot representation area and, as a re- 
sult, a midcentral focused ERD can be found. Such 
an ERD pattern is, however, relatively rare since the 
location of the foot representation area is in the 
mesial brain surface (Ikeda, Luders, Burgess, & 
Shibasaki, 1992) and the potentials of which are 
not easily accessible by EEG electrodes. In contrast 
to this rare midcentral ERD, lateralized ERS (in 
both hand representation areas) is frequently ob- 
served (Neuper & Pfurtscheller, 2001). To quantify, 
five out of nine (trained) able-bodied subjects ex- 
hibited hand-area mu ERS during a foot motor im- 
agery task. One typical example is shown on the 
right hand side of figure 20.3. Foot motor imagery 
desynchronized the foot-area mu rhythm and en- 
hanced the hand-area mu rhythm in both hemi- 
spheres. This can be interpreted such that foot 
motor imagery activates not only the corresponding 
representation area but simultaneously deactivates 



(inhibits) networks in the hand representation area 
and synchronizes the hand-area mu rhythm. This is 
very important in BCI research, because a motor- 
imagery-induced synchronization of sensorimotor 
rhythms is an important feature that is used to ob- 
tain high classification accuracy. Further support for 
this idea of intramodal interaction is derived from 
PET experiments, where a decrease in blood flow 
has been observed in the somatosensory cortical 
representation area of one body part (e.g., hand 
area), whenever attention is diverted to a distant 
body part (e.g., foot area; Drevets et al, 1995). 

For comparison, the data obtained during 
hand motor imagery in the same subject are also 
displayed on the left side of figure 20.3. As ex- 
pected, hand motor imagery induced a marked de- 
synchronization (ERD) at the electrodes overlaying 
the hand representation area and a synchronization 
(ERS) at parieto-occipital electrodes. This pattern, 
central ERD and parieto-occipital ERS, can be seen 
as an example of an intermodal interaction be- 
tween motor and extrastriate visual areas, charac- 
teristic of an activation of hand-area networks and 
deactivation or inhibition of parieto-occipital net- 
works. In this respect, it is of interest to refer to the 
work of Foxe, Simpson, and Ahlfors (1998). They 
reported an increase of parieto-occipital alpha 
band activity when the subject was engaged in an 
auditory attention task. This would indicate an ac- 
tive mechanism for suppressing stimulation in the 
visual processing areas. 
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Figure 20.3. Intermodal focal ERD/surround ERS during hand motor imagery (left) and intramodal focal 
ERD/surround ERS during foot motor imagery (right). Displayed are band power time courses (ERD/ERS) +95% 
confidence intervals for selected electrode positions. The dashed line corresponds to the cue onset. ERD, event- 
related desynchronization; ERS, event-related synchronization. 



Synchronization of Central 
Beta Oscillations 

Imagination of foot movement not only synchro- 
nizes the hand-area mu rhythm but also produces a 
beta rebound at the vertex in the majority of sub- 
jects. Of nine able-bodied subjects, seven were 
found to exhibit such a beta rebound (second and 
third columns, table 20.1). Both a strict localiza- 
tion at the vertex and the fact that the most reactive 
frequency components were found in the 25-35 Hz 
band characterized this beta rebound. Such a mid- 
centrally induced beta rebound after hand motor 
imagery had been reported earlier (Pfurtscheller & 
Lopes da Silva, 1999), but it was found to be spo- 
radic by comparison to the beta rebound seen after 
both feet imagery. 

We hypothesize that the occurrence of motor 
cortex activity, independent of whether it follows 
the actual execution or just imagination of a move- 
ment, may involve at least two networks, one cor- 
responding to the primary motor area and another 
one in the supplementary motor area (Pfurtscheller 
et al., 2005). Imagination of both feet movements 
may involve both the supplementary motor area 
and the two cortical foot representation areas. Tak- 
ing into consideration the close proximity of these 
cortical areas (Ikeda et al, 1992), along with the 
fact that the response of the corresponding net- 



works in both areas may be synchronized, it is 
likely that a large-amplitude beta rebound occurs 
after foot motor imagery. Because of its limited fre- 
quency band (between 20 and 35 Hz) and its large 
magnitude at the vertex, this beta rebound feature 
is a good candidate for obtaining a high classifica- 
tion accuracy in single-trial EEG classification. In a 

Table 20.1. Percentage Band Power Increase 
and Beta Rebound 
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Note. Percentage band power increase and reactive frequency 
band after termination of foot motor imagery referred to a 1-second 
time interval before the motor imagery task (left side). Results of 
single-trial classification of the beta rebound using Distinction Sen- 
sitive Learning Vector Quantization (Pregenzer &r Pfurtscheller, 
1999) and frequency band with the highest classification accuracy 
(right side). Only one EEG channel was analyzed, recorded at elec- 
trode position Cz. Modified from Pfurtscheller et al., 2005. 
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pilot study, classification accuracies between 59% 
and 89% in seven out of nine healthy subjects were 
achieved when only one EEG channel (electrode 
position Cz) was classified against rest (fourth and 
fifth columns, table 20.1). 



BCI Training 

BCI Training with Feedback 

The enhancement of oscillatory EEG activity (ERS) 
during motor imagery is a very important aspect of 
BCI research and presumably requires positive rein- 
forcement. Feedback regulation of the sensorimotor 
oscillatory activity was originally derived from ani- 
mal experiments, where cats were rewarded for pro- 
ducing increases of the sensorimotor rhythm (SMR; 
Sterman, 2000). It has been documented over many 
years that human subjects can also learn to enhance 
or to suppress rhythmic EEG activity when they 
are provided with information regarding the EEG 
changes (e.g., Mulholland, 1995, Neuper, Schlogl, 
& Pfurtscheller, 1999; Sterman, 1996; Wolpaw & 
McFarland, 1994; Wolpaw, McFarland, Neat, & 
Forneris, 1991). The process of acquiring control of 
brain activity (i.e., to deliberately enhance patterns 
of oscillatory activity) can therefore be conceptual- 
ized as an implicit learning mechanism involving, 
among other processes, operant learning. 

The main rationale of (classifier-based) BCI 
training is, however, to take advantage of both the 
learning progress of the individual user and, simul- 
taneously, the learning capability of the system 
(Pfurtscheller & Neuper, 2001). This implies that 
two systems (human being and machine) have to 
be adapted to each other simultaneously to achieve 
an optimal outcome. Initially, the computer has to 
learn to recognize EEG patterns associated with 
one or more states of mental imagery. This implies 
that the computer has to be adapted to the brain 
activity of a specific user. After this phase of ma- 
chine learning, when an appropriate classifier is 
available, the online classification of single EEG 
trials can start and feedback can be provided to 
enable the user's learning, thereby enhancing the 
target EEG patterns. As a result of feedback train- 
ing, the EEG patterns usually change, but not 
necessarily in the desired direction (i.e., diver- 
gence may occur). For this reason, the generation 
of appropriate EEG feedback requires dynamic 



adjustment of the classifier and of the feedback 
parameters. 

To keep the training period as short as possi- 
ble, a well thought-out training paradigm is neces- 
sary. In this context, two aspects are crucial: (a) the 
exact manner of how the brain signal is translated 
into the feedback signal (i.e., information content 
of the feedback; for advantages of providing 
continuous or discrete feedback, see McFarland, 
McCane, & Wolpaw, 1998; Neuper et al., 1999); 
and (b) the type of feedback presentation (i.e., vi- 
sual versus auditory feedback; abstract versus 
concrete or realistic feedback). In any case, the in- 
fluence of the feedback on the capacity for atten- 
tion, concentration, and motivation of the user, all 
aspects that are closely related to the learning pro- 
cess, should be considered. In general, it is impor- 
tant to design an attractive and motivating feedback 
paradigm. One such example, the so-called basket 
paradigm, is described in the following section. 

BCI Training with a Basket Paradigm 

Motivation is crucial for the learning process. The 
same applies to BCI feedback training. Users have to 
learn to control their own brain activity by reliably 
generating different brain patterns. For this purpose, 
a simple gamelike feedback paradigm was imple- 
mented. The object of the game is to move a falling 
ball horizontally to hit a highlighted target (basket) 
at the base of a computer monitor. The horizontal 
position of the ball is controlled by the user's mental 
activity (BCI classification output), whereas the ve- 
locity of fall (defined by the trial length) is constant. 
Even though the graphical representation of the par- 
adigm is deliberately simple, like those of the first 
computer games, users achieved good BCI control 
within a short period of time. The results of a study 
including four young paraplegic patients showed 
that three out of the four had satisfying results after 
a few runs using the basket paradigm (Krausz, 
Scherer, Korisek, & Pfurtscheller, 2003). Predefined 
electrode positions (C3 and C4 according to the 
international 10/20 system) and standard alpha 
(10-12 Hz) and beta (16-24 Hz) frequency bands 
were used to generate the feedback. A Butterworth 
filter and simple amplitude squaring was applied for 
the band power estimation of the acquired EEG sig- 
nal (analog band pass between 0.5 and 30 Hz, 
sample rate 128 Hz). The feature values used for 
classification were calculated by averaging across a 
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1 -second time window that was shifted sample-by- 
sample along the band power estimates. The pa- 
tients had the task of landing the ball on the basket 
(situated on either the left or right half of the base of 
the screen). For this two-class problem, an LDA 
classifier was used. The classifier was computed 
by analyzing (10 x 10 cross-validation) cue-based 
motor-imagery-related EEG patterns collected dur- 
ing a motor imagery assessment session at the be- 
ginning of the study: The patients were required to 
imagine the execution of movements corresponding 
to each of a sequence of visual cues presented. By 
analysis of the resulting brain patterns for each sub- 
ject, the most suited (discriminable) motor imagery 
tasks were found, and the classifier output (position 
of the ball) was weighted to adjust the mean deflec- 
tion to the middle of the target basket. In this way, 
the BCI output was uniquely adapted to each pa- 
tient. It was found that online classification accura- 
cies of 85% and above could be achieved in a short 
time (3 days, 1.5 hours per day). 



BCI Application for Severely 
Paralyzed Patients 

Completely paralyzed patients without any con- 
scious control of muscle activity can only commu- 
nicate with their environment when, through the 
use of EEG signals, an electronic spelling system is 
controlled. There is evidence that a BCI based on 
oscillatory EEG changes, induced by motor im- 
agery, can be utilized to restore communication in 
severely disabled people (Neuper, Muller, Kubler, 
Birbaumer, & Pfurtscheller, 2003). Even completely 
paralyzed patients, who had lost all voluntary mus- 
cle control, learned to control how to enhance or 
suppress specific frequency components of the 
sensorimotor EEG by using a motor imagery strat- 
egy. In order for such a patient to obtain control 
over brain oscillations, BCI training sessions have 
to be conducted regularly (i.e., 2 times a week) 
over a period of several months. 

Here we report, as an example, the case of a 
male patient (60 years old) who suffered from ALS 
for more than 5 years, being artificially ventilated. 
This patient was totally paralyzed and had almost 
lost his ability to communicate altogether. Initially, 
the patient was trained to produce two distinct 
EEG patterns by using an imagery strategy. For 
this, the so-called basket paradigm, as described 



above (for details, see Krausz et al, 2003), was em- 
ployed. It requires continuous (one-dimensional) 
cursor control to direct a falling ball on a computer 
monitor into one of two baskets (the target is indi- 
cated in each trial). The EEG signal (band pass 
5-30 Hz, sampling rate 128 Hz) used for classifica- 
tion and feedback was initially recorded from three 
bipolar channels over the left and right sensori- 
motor areas and close to the vertex. To generate 
the feedback, two frequency bands were used 
(8-12 Hz and 18-30 Hz). The feedback was calcu- 
lated by a linear discriminant classifier, which was 
developed to discriminate between two brain states 
(Pfurtscheller, Neuper, Flotzinger, & Pregenzer, 
1997). In 17 training days, the patient performed 
82 runs with the basket paradigm. The effective- 
ness of the training is suggested by the significant 
increase of the classification accuracy from random 
level at the beginning (mean over 23 runs, per- 
formed on the first two training days) to an average 
classification rate of 83% (mean over 22 runs, per- 
formed on the last six training days). At the end of 
the reported training period, this patient was able 
to voluntarily produce two distinct EEG patterns, 
one being characterized by a broad-banded desyn- 
chronization and the other by a synchronization in 
the form of induced oscillations (ERS) in the alpha 
band (see figure 20.4). This EEG control enabled 
the patient to later use the so-called virtual key- 
board (VK; Obermaier, Muller, & Pfurtscheller, 
2003) for copy spelling of presented words and, fi- 
nally, for free spelling of a short message. Since the 
design of a convenient BCI spelling system is still a 
matter of current research, the following section 
provides a short overview of different approaches. 



BCI-Based Control of 
Spelling Systems 

The VK spelling system presented in this section is 
an important application of the Graz-BCI (Guger 
et al., 2001; Pfurtscheller & Neuper, 2001; Scherer, 
Schlogl, Muller-Putz, & Pfurtscheller, 2004). As de- 
scribed above, it is based on the detection and clas- 
sification of motor imagery-related EEG patterns. 

The basic VK allows the selection of letters from 
an alphabet by making a series of binary decisions 
(Birbaumer et al., 1999; Obermaier et al., 2003). 
This means that the user has to learn to reliably 
reproduce two different EEG patterns (classes). 
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Figure 20.4. Picture of a patient suffering from ALS during feedback training. In the upper right corner, the bas- 
ket paradigm is shown (left). Examples of trials and event-related desynchronization and event-related synchro- 
nization time/frequency maps display ipsilateral ERS (right). See also color insert. 



Starting with the complete alphabet displayed on a 
screen, subsets of decreasing size are successively se- 
lected until the desired letter is one of two options. 
A dichotomous structure with five consecutive lev- 
els of selection (corresponds to 2 5 = 32 different let- 
ters) and two further levels of confirmation (OK) 
and correction (BACK/DEL) is implemented (see 
figure 20.5, top left). BACK allows a cancellation of 
the previous subset selection by returning to the 
higher selection level, while choosing DEL results in 
deletion of the last written letter. A measure for the 
communication performance is the spelling rate o, 
given as correctly selected letters per minute. A 
study using healthy subjects showed that spelling 
rates between 0.67 and 1.02 letters per minute could 
be achieved when using a trial length of 8 seconds. 
Six correct selections are required per letter, resulting 
in a maximum o = (60 sec/6.8 sec) = 1.25 letters per 
minute (Obermaier et al., 2003). A bar, either on the 
left- or right-hand side of the screen, was presented 
as feedback (figure 20.5, top right). Its purpose was 
to indicate either the first or the second subset. The 
subjects were required to spell predefined words by 
selecting the appropriate letter subgroup by motor 
imagery. Hidden Markov models (Rabiner & Juang, 
1986) were used to characterize the motor imagery- 
related EEG and classification based on maximum 
selection of likelihood. 

The dichotomous selection strategy is a simple 
but efficient method to make a single choice (letter) 
from a number of items (alphabet). The selection of 
1 item from a set of 32 items requires only five bi- 



nary selections. Although the achieved spelling 
rates are low and do not reflect an appropriate com- 
munication speed, nonneuromuscular interaction 
with one's environment is possible. A major prob- 
lem of this method, however, comes with a misclas- 
sification. One false selection requires, depending 
on the selection level, up to 13 correct selections in 
order to cancel the mistake. Given the fact that sub- 
jects usually start with a classification rate of 70% 
(which means that 3 out of 10 EEG patterns are 
misclassified, thereby greatly decreasing the com- 
munication rate), it is clear that training to achieve a 
high classification accuracy and consequently a 
good adaptation between the subject and the BCI is 
necessary. For the majority of the subjects, a longer 
training period results in better BCI performance. 

There is also potential to increase the spelling 
rate in other ways. One possibility is the use of word 
prediction. The prediction can be based on consid- 
ering language-specific probability occurrences of 
letters (e.g., the letter e has the highest probability of 
occurrence in the German and English languages) 
and their intra- and interword position. The predic- 
tion can also be based on a predefined dictionary 
such as the T9 (Tegic Communications, Inc.) pre- 
dictive text input system (currently implemented in 
many mobile phones). A modified version based on 
eight keys was implemented for the two-class cue- 
based VK. Again, a dichotomous structure such as 
that described above is used (figure 20.5, bottom 
left). With this system, we demonstrated that healthy 
subjects could achieve spelling rates (a) of up to 
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Figure 20.5. Dichotomous letter selection structure (top left). Screen shots of the basic two-class 
virtual keyboard. Four of the six steps to perform to select a letter are shown (top right). Letter 
selection structure based on T8 (bottom left). Three-class asynchronously controlled virtual key- 
board. Example: Selection of the letter H by virtual keyboard (bottom right). See also color insert. 



4.24 letters per minute when a dictionary contain- 
ing 145 words was used. 

A further possibility to speed up communica- 
tion performance is to use more than two reliably 
detectable brain patterns. If the two-class proce- 
dure described above is divided into three instead 
of two parts, fewer selection steps are necessary. A 
decrease of the trial length would also increase the 
communication performance. 

Another important factor is classification accu- 
racy. The information transfer rate 

ITR [bit/min] 

= (log 2 N+Plog 2 P 



+ (l-P)log 



2 N-ly trial length 



60 



describes the relationship between number of 
classes (N), classification accuracy (P), and trial 
length (Wolpaw et al., 2000). A large N, a high P, 
and a short trial length result in an increased ITR. 



A VK based on three classes that additionally 
uses the asynchronous operating mode has been 
introduced (Scherer, Muller, Neuper, Graimann, & 
Pfurtschdler, 2004). The asynchronous mode frees 
subjects from the constraints of a fixed timing 
scheme. Whenever the subject is ready to generate 
a motor imagery-related brain pattern, the BCI is 
able to detect and process it. The new operation 
mode also allows a new VK design. The screen is di- 
vided into two parts: A small upper part displays 
the selected letters and words, while the letter selec- 
tion process and the visual biofeedback (moving 
cursor) take up the remaining lower part of the 
screen. The letters of the alphabet are arranged al- 
phabetically on two vertically moving assembly 
lines on the left and right half of the screen. A verti- 
cal displacement between the letters on both sides 
should avoid the sensation of competition between 
the single letters. Every five letters, a control com- 
mand can be selected: DEL, used to delete the last 
spelled letter, and OK to confirm the spelled word. 
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At all times, five items are visible on each side. As 
long as the first of the three brain patterns is 
detected, the items on both sides of the screen scroll 
from the bottom to the top. If an item reaches the 
top of the selection area, it disappears and the next 
one appears from the bottom. In order to avoid dis- 
turbing influences and give subjects the opportu- 
nity to concentrate on the moving objects, the 
feedback cursor was hidden during the scrolling 
process. The item at the topmost position can be se- 
lected by moving the feedback cursor toward the 
desired left or right direction by performing the sec- 
ond or third motor imagery-related brain pattern. 
An item is selected if the cursor exceeds a subject- 
specific left- or right-side threshold for a subject- 
specific time period. Selected letters are presented 
in the upper part of the screen and the spelling pro- 
cedure starts over again. If none of the trained brain 
patterns is detected, the VK goes into the standby 
mode (see figure 20.5, bottom right). A study with 
healthy subjects revealed spelling rates (a) of up to 
3.38 letters per minute (mean 1.99 letters per 
minute). Compared to the standard two-class VK, 
double in performance could be achieved. Another 
outcome of the study was the conclusion that not 
every subject was able to control the application, 
even if the classification accuracy was higher than 
90%. A possible explanation for this may be the 
short period available for the training. Three classes 
and asynchronous control seems to be more de- 
manding than the synchronous two-class control 
mode. Also, the dynamic scenario of many moving 
objects may have disturbing influences and cause 
changes to, or even a deterioration of, the EEG. The 
BCI analyzes the ongoing EEG by estimating the 
band power in subject-specific bands (optimized 
by means of a genetic algorithm; Holland, 1975). 



Three LDA classifiers were trained to discriminate 
in each case between two out of the three classes. 
The overall classification output was generated by 
combining the results of the three LDA classifiers 
(majority voting). 



BCI-Based Control of Functional 
Electrical Stimulation in 
Tetraplegic Patients 

For patients with high spinal cord injury, the 
restoration of grasp function by means of an orthosis 
(Pfurtscheller et al, 2000) or neuroprostheses 
(Heasman et al., 2002; Pfurtscheller et al., 2003) has 
developed into an important area of application. 

We had the opportunity to work with a 
tetraplegic patient (complete motor and sensory le- 
sion below C5) for several months. During this time, 
the patient learned to reliably induce 17 Hz oscilla- 
tions in EEG (modulated by foot motor imagery) 
recorded from electrode position Cz. By estimating 
the band power in the frequency band between 15 
and 19 Hz and by applying simple threshold classi- 
fication, the foot motor imagery brain pattern can be 
detected with an accuracy of 100% (see figure 20.6). 
Since the patient learned to generate the brain pat- 
tern at will, the asynchronous operating mode could 
be used to control a neuroprosthesis based on sur- 
face functional electrical stimulation (Pfurtscheller 
et al., 2003). Every time the patient would like to 
grasp an object, he can consecutively initiate 
("brain-switch") phases of the grasp. In the case of a 
lateral grasp (e.g., to use a spoon), three phases have 
to be executed in this order: (1) finger and thumb 
extension (hand opens); (2) finger flexion and 
thumb extension (fingers close); and (3) finger and 
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Figure 20.6. Logarithmic band power (15-19 Hz) with threshold used for detection and the phases of the 
lateral grasp (left). Picture of the patient with surface functional electrical stimulation (right). 
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thumb flexion (thumb moves against closed fin- 
gers). In this way, the quality of life of the patient 
improved dramatically because of the independence 
gained (Pfurtscheller et al., 2005). Another patient 
(complete motor and sensory lesion below C5) with 
whom we had the opportunity to work had a neuro- 
prosthesis (Freehand system; Keith et al., 1989) im- 
planted in his right arm. During a three-day training 
program, the patient learned to generate two differ- 
ent brain patterns. First the patterns were reinforced 
using the cue-based basket paradigm, and then dur- 
ing free training the patient learned to produce the 
patterns voluntarily (asynchronous mode). In the 
latter case, the feedback was presented in the form 
of a ball, placed in the middle of the screen, which 
could be moved either to the left or to the right side. 
The EEG signals (C3 and Cz) were analyzed contin- 
uously and fed back to the patient. After the short 
training period, the patient was able to operate the 
neuroprosthesis by mental activity alone. These re- 
sults show that a BCI may also be an alternative ap- 
proach for clinical purposes. 



Perspectives for the Future 

User acceptance of a BCI depends on its reliability 
in obtaining a high information transfer rate (ITS) 
within a minimum number of training sessions and 
its practicability of application (e.g., reduction of 
number of electrodes required). The ITS can be in- 
creased by a corresponding increase in the number 
of recognizable mental tasks and also by achieving 
BCI operation in an asynchronous mode. A realis- 
tic projection for the future would see three or four 
mental tasks with noninvasive EEG recordings be- 
ing utilized, where the type and optimal combina- 
tion of mental tasks would have to be selected in 
separate sessions for each user. To obtain high ac- 
curacy in single trial classifications, it is important 
not only to use powerful feature extraction meth- 
ods but also to search for spatiotemporal EEG pat- 
terns displaying task-related synchronization or 
ERS. Recognition of such ERS phenomena is a pre- 
requisite for a high hit rate and a low false positive 
detection rate in an asynchronous BCI. 

The ITS can be increased when subdural ECoG 
recordings are available. In this case, the signal-to- 
noise ratio is much higher than that of the EEG, 
and high-frequency components such as gamma 
band oscillations can be used for classification. 



ECoG recordings in patients therefore hold great 
promise, and their utilization is an immediate chal- 
lenge for the near future. 

When the EEG is used as the input signal for a 
BCI system, multichannel recordings and special 
methods of preprocessing (i.e., independent com- 
ponent analysis, ICA) are recommended. Sensori- 
motor rhythms such as mu and central beta are 
particularly suitable for ICA because both are spa- 
tially stable and can therefore be separated easily 
from other sources (Makeig et al., 2000). More ex- 
tensive research is also needed in order to extract 
as many possible features from the EEG, from 
which a small number may be selected to optimize 
the quality of the classification system. For this fea- 
ture selection, different algorithms have been pro- 
posed such as distinction sensitive learning vector 
quantification, an extension of Kohonen's learning 
vector quantification (Pregenzer & Pfurtscheller, 
1999) and the genetic algorithm (Graimann, Hug- 
gins, Levine, & Pfurtscheller, 2004). 

More extensive work is also needed to specify 
the mental task and to optimize the feedback pre- 
sentations. So, for example, hand motor imagery 
can be realized either by visualization of one's own 
or another's hand movement (visuomotor imagery) 
or by remembering the feeling of hand movement 
(kinesthetic motor imagery; Curran & Stokes, 
2003). Both types of mental strategies result in dif- 
ferent activation patterns and have to be investi- 
gated in detail. The feedback is important to speed 
up the training period and to avoid fatigue by the 
user. The importance of visual feedback and its im- 
pact on premotor areas was addressed by Rizzo- 
latti, Fogassi, and Gallese (2001). 

EEG recording with subdural electrodes is an in- 
vasive method ol importance in neurorehabilitation 
engineering (Levine et al., 2000). The advantage of 
the ECoG over the EEG is its better signal-to-noise 
ratio and therewith also the easier detection of, for 
example, gamma activity. Patient-oriented work with 
subdural electrodes and ECoG single trial classifica- 
tion have shown the importance of gamma activity in 
the frequency band of 60-90 Hz (Graimann, Hug- 
gins, Schlogl, Levine, & Pfurtscheller, 2003). 

A number of studies in monkeys have demon- 
strated that 3-D control is possible when multiunit 
activity is recorded in cortical areas and firing pat- 
terns are analyzed (Serruya, Hatsopoulos, Panin- 
ski, Fellows, & Donoghue, 2002; Wessberg et al., 
2000). This technique is highly invasive and based 
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on the recording from multiple electrode arrays, 
each of which with up to 100 sensors implanted 
in different cortical motor areas (Maynard et al., 
1997). 
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Main Points 

1 . The preparatory phase of an actual movement 
and the imagination of the same movement 
(motor imagery) can modify sensorimotor 
rhythms in a very similar way 

2. Motor imagery-induced changes in oscillatory 
electroencephalogram activity can be 
quantified (event-related desynchronization, 
ERD/ERS), classified, and translated into 
control commands. 

3. The mutual adaptation of machine and user, as 
well as feedback training, is crucial to achieve 
optimal control. 

4. Existing applications show that BCI research is 
ready to leave the laboratory and prove its 
suitability in assisting disabled people in 
everyday life. 
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Artificial Vision 



The main guideline of research in neuroergonom- 
ics is to find out how the brain copes with the mas- 
sive amount and variety of information that must 
be processed to carry out complex tasks of every- 
day life (Parasuraman, 2003). From the viewpoint 
of visual neuroscience, our primary attention is ad- 
dressed to the question of how the brain encodes 
visual information and how we can learn more 
about the neural mechanisms underlying vision. 
Such a pursuit is relevant not only to developing 
an understanding of visual physiology per se, but 
also to providing insight into environmental modi- 
fications to assist and enhance human perfor- 
mance. Our long-term goal is to learn how to "talk" 
to the brain — and create vision where the eye 
and the brain fail, so that we might be able to im- 
prove the quality of life for some visually impaired 
patients. 

Based on the dramatic evolution of microtech- 
nology over the past decades, it is now possible to 
create devices, that is, neuroprostheses, that have 
the potential to restore functions even though ele- 
ments of the sensory or motor systems are dam- 
aged or lost, for example, due to a disease process 
like macular degeneration, or due to lesions in- 
duced by surgery, brain trauma, or stroke. In the 
auditory domain, the attempt to artificially stimu- 



late peripheral parts of the pathway in deaf individ- 
uals by means of a cochlear implant has proven 
highly successful. Cochlear implants are capable of 
restoring hearing and providing language compre- 
hension in patients (Copeland & Pillsbury, 2004). 
Although knowledge about the structure and func- 
tion of the visual system is more advanced than in- 
formation on the auditory system, the goal of 
creating a comparable prosthesis for blind patients 
has yielded only moderate success so far (see next 
section). The challenge in designing retinal im- 
plants or visual cortical stimulators is not related 
to the manufacture or implantation of such a 
device — this has been done already by several 
groups. Rather, the most serious challenge relates 
to the need to construct devices that talk to the vi- 
sual brain in a way that makes sense. 

Thus, visual neuroscience and the neuroer- 
gonomics of artificial vision are not confined to 
matters of engineering and must include inquiries 
to increase our understanding of the anatomy and 
physiology of the visual pathway. We are using a 
systemic approach to make an effort to understand 
the language of the brain. 

In this chapter, we present an overview of re- 
search on retinal prostheses and progress in this 
field and then provide a perspective on artificial 



329 



330 



Special Populations 



vision and the role of modern technologies to sup- 
port research in this sector. We also review studies 
on brain plasticity that provide essential informa- 
tion on changes in the brain as a consequence of 
(partial) blindness, the interaction of visual areas 
with other sensory modalities, and possible ways of 
influencing brain plasticity as a basis of visual reha- 
bilitation. 



History of Visual Prostheses 

The primary impetus to develop a visual prosthesis 
stems from the fact that blindness affects millions 
of people worldwide. Further, there are no effective 
treatments for the most incapacitating causes of 
blindness. In response to this large unmet need, 
many groups have pursued means of artificially 
restoring vision in the blind. It has been known 
for decades that electrical stimulation delivered 
to intact visual structures in a blind individual 
can evoke patterned sensations of light called 
phosphenes (Go the, Brandt, Irlbacher, Roricht, 
Sabel & Meyer, 2002; Marg & Rudiak, 1994). It 
has therefore been assumed that if electrical stim- 
ulation could be somehow delivered in a con- 
trolled and reproducible manner, patterns encoding 



meaningful shapes could be generated in order to 
potentially restore functional vision. With this 
foundation and with roughly three decades of 
work on visual prostheses, significant advances 
have been made in a relatively short time (for 
review, see Loewenstein, Montezuma, & Rizzo, 
2004; Margalit et al, 2002; Maynard, 2001; Mer- 
abet, Rizzo, Amedi, Somers, & Pascual-Leone, 
2005; Rizzo et al., 2001; Zrenner, 2002). 

One of the earliest attempts to develop a visual 
prosthetic was made by applying electrical stimula- 
tion to the visual cortex (figure 2 1.1 A). In one 
profoundly blind patient, electrical stimulation de- 
livered to the surface of the brain allowed the pa- 
tient to report crude phosphenes that were at least 
in spatial register with the known cortical retino- 
topic representation of visual space (Brindley & 
Lewin, 1968). More recent efforts have incorpo- 
rated a digital video camera mounted onto a pair of 
glasses interfaced with a cortical-stimulating array 
that decodes an image into appropriate electrical 
stimulation patterns (Dobelle, 2000). Although the 
cortical approach has provided an important foun- 
dation in terms of feasibility, several technical chal- 
lenges and concerns remain, including the sheer 
complexity of the visual cortex and the invasive- 
ness of surgical implantation. 



A. Cortical Approach 



B. Retinal Approach 





Figure 21 .1 . Schematic diagram of a cortical (A) and retinal (B) approach to restore vision. 
In the cortical approach, an image captured by a camera (not shown) is translated into an ap- 
propriate pattern of cortical stimulation to generate a visual precept. Inset figure shows a 100 
microelectrode array that could be used to stimulate the cortex (Utah Array, modified from 
Maynard, Nordhausen, & Normann, 1997). In the retinal approach, an electrode array 
(shown in inset) can be placed directly on the retinal surface (epiretinal) or below (subreti- 
nal) to stimulate viable retinal ganglion cells. The retinal prosthesis can stimulate by the 
power captured by incident light or use signals received from a camera and signal processor 
mounted externally on a pair of glasses (not shown). 
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Figure 21 .2. Geometries of some subretinal and epiretinal prosthetic devices currently being pursued. (A) Schematic drawing of a 
subretinal implant placed under a retina with photoreceptor degeneration. The small electrodes on the implant are designed to stimu- 
late inner retinal cells. (B) Epiretinal implant with extraocular components. A primary coil on the temple transmits to a secondary coil 
on the sclera. A cable carries power and signal to a stimulator chip, which distributes energy appropriately to electrodes on the epiret- 
inal surface. (C) Epiretinal implant with intraocular components. A camera on a spectacle frame provides the signal. A primary coil in 
the spectacles transmits to a secondary coil in an intraocular lens. A stimulator chip distributes power to multiple electrodes on the 
epiretinal surface. From Loewenstein, Montezuma, and Rizzo (2004). Copyright 2004 American Medical Association. 
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An alternative approach is to implant the pros- 
thesis at a more proximal point of the visual path- 
way, namely the retina (figure 2 LIB). In retinitis 
pigmentosa and age-related macular degeneration, 
two disorders that contribute greatly to the inci- 
dence of inherited blindness and blindness in the 
elderly (Hims, Diager, & Inglehearn, 2003; Klein, 
Klein, Jensen, & Meuer, 1997), there is a relatively 
selective degeneration of the outer retina, where 
the photoreceptors lie. Ganglion cells, which lie on 
the opposite side of the retina and connect the eye 
to the brain, survive in large numbers and are able 
to respond to electrical simulation even in highly 
advanced stages of blindness (Humayun et al., 
1996; Rizzo, Wyatt, Loewenstein, Kelly, & Shire, 
2003a, 2003b; see also Loewenstein et al., 2004, 
for review). Thus, one could potentially implant a 
prosthetic device at the level of the retina to stimu- 
late the middle and inner retina (ganglion and 
bipolar cells) and potentially replace lost photore- 
ceptors. Given that ganglion cells are arranged in 
topographical fashion throughout the retina, the 
generation of a visual image can theoretically be 
made possible by delivering multisite patterns of 
electrical stimulation. Two methods are being pur- 
sued that differ primarily with respect to the loca- 
tion at which the device interfaces with the retina 
(see figure 21.2). A subretinal implant is placed be- 
neath the degenerated photoreceptors by creating a 
pocket between the sensory retina and retinal pig- 
ment epithelium. Alternatively, an epiretinal im- 
plant is attached to the inner surface of the retina 
in close proximity to the ganglion cells. Both de- 
vices require an intact inner retina to send visual 
information to the brain and thus are not designed 
to work in conditions where the complete retina or 
optic nerve have been compromised (e.g., as might 
occur in diabetic retinopathy, retinal detachment, 
or glaucoma). 

The subretinal implant design is being pursued 
by several groups of investigators, including our 
Boston-based project (see also Chow et al, 2004; 
Zrenner et al., 1999). In one design, natural inci- 
dent light that falls upon a subretinal photodiode 
array generates photocurrents that stimulate retinal 
neurons in a 2-D spatial profile. Such devices can 
easily be made with hundreds or thousands of neu- 
rons. Chow and coworkers have carried out a 
phase I feasibility trial in six patients with pro- 
found vision loss from retinitis pigmentosa. Pa- 
tients were followed from 6 to 18 months after 



implantation and reported an improvement in vi- 
sual function. This was documented by an increase 
in visual field size and the ability to name more let- 
ters using a standardized visual acuity chart (Chow 
et al., 2004). However, these results have met with 
controversy. It seems that the purported beneficial 
outcome might not be the result of direct and pat- 
terned electrical stimulation as initially anticipated 
but instead an indirect "cell rescue" effect from 
low-level current generated by the device (Pardue 
etal.,2005). 

Concurrently, large efforts have pursued the 
epiretinal approach (Rizzo et al., 2003a, 2003b; 
Humayun et al., 2003). Much like the cortical ap- 
proach (and unlike the subretinal approach), the 
current epiretinal strategies incorporate a digital 
camera mounted on a pair of eyeglasses to capture 
an image that in turn is converted into electrical 
signals. Both short-term and long-term testing in 
human volunteers with advanced RP has been car- 
ried out (Rizzo et al., 2003a, 2003b; Humayun 
et al., 1996, 2003). Intraoperative experiments that 
lasted only minutes to hours while patients re- 
mained awake produced visually patterned percep- 
tions that were fairly crude. As predicted, the gross 
geometric structure of the phosphene patterns 
could be altered to some extent by varying the po- 
sition and number of the stimulating electrodes 
and the strength or duration of the delivered cur- 
rent (Rizzo et al., 2003a, 2003b). Long-term hu- 
man testing has only been performed by Humayun 
and coworkers, who have permanently implanted 
an epiretinal prosthesis in roughly six blind pa- 
tients. The implanted device included an intraocu- 
lar electrode array of platinum electrodes arranged 
in a 4 x 4 matrix. The array is designed to interface 
with the retina and camera connected to image- 
processing electronics. Their first subject has re- 
ported seeing perceptions of light (phosphenes) 
following stimulation of any of the 16 electrodes of 
the array. In addition, the subject is able to use im- 
ages captured with the camera to detect the pres- 
ence or absence of ambient light, to detect motion, 
and to recognize simple shapes (Humayun et al., 
2003). 

The results from these research efforts are en- 
couraging and demonstrate (at least in principle) 
that patterned electrical stimulation can evoke pat- 
terned light perceptions. However, the perceptual 
pattern often does not match the stimulation pattern 
(Rizzo et al., 2003a, 2003b). The inconsistencies 
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and limitations of the results may relate to the fact 
that stimulation is carried out on severely degener- 
ated and therefore physiologically compromised 
retinas. To date, a key milestone has yet to be 
achieved: the demonstration that a visual neuro- 
prosthesis can improve the quality of life of these 
individuals by allowing truly functional vision, 
such as the recognition of objects or even skillful 
navigation in an unfamiliar environment. 

We believe that these engineering and surgical 
issues no longer represent the greatest barriers to 
future progress. Rather, the greatest limitation to 
the effective implementation of this technology is 
our own ignorance about how to communicate vi- 
sual information to the brain in a meaningful way 
(Merabet et al., 2005). Simple generation of crude 
patterns of light will not likely be very helpful to 
blind patients when they ambulate in an unfamiliar 
environment. However, we appreciate that even 
this meager success may provide useful cues in a 
familiar environment, like sitting at one's own 
kitchen table. Furthermore, increasing image reso- 
lution (e.g., by increasing the number of stimulat- 
ing electrodes) with the goal of generating more 
complex perceptions may or may not provide more 
useful visual percepts — the potential for enhanced 
success depends upon our ability to communicate 
with the brain. We propose that a deeper under- 
standing of how the brain adapts to the loss of 
sight and how the other remaining senses process 
information within the visually deprived brain is 
necessary to make it more likely that we will be 
able to restore vision with a neuroprosthetic de- 
vice. 



ate alternatives to the natural pathways. Incorpo- 
rating this broad approach, we believe, will make it 
more likely that we will be able to create useful 
prostheses for blind patients. By useful, we mean to 
imply that our assistive technology will be able to, 
at the least, help blind patients navigate in unfamil- 
iar environments. 

The particular tools used for investigating vi- 
sual information processing are crucial for the im- 
plementation of such an approach. Over the past 
decades, the development of imaging and stimula- 
tion methods has provided a toolbox for neurosci- 
entists that allows the noninvasive monitoring of 
brain functions and their changes over time. Using 
these techniques, we can identify not only where in 
the brain a process takes place (functional Mag- 
netic Resonance Imaging, (fMRI), but also when or 
in which sequence it occurs (Electroencephalogra- 
phy, EEG, or Magnetoencephalography, MEG), and 
whether a part of the brain is causally related to the 
process in question (Transcranial Magnetic Stimu- 
lation, TMS; lesion studies, Kobayashi & Pascual- 
Leone, 2003; Merabet, Theoret, & Pascual-Leone, 
2003; Rossini & Dal Forno, 2004). In this section, 
we give a short introduction to these technologies 
and describe how they can be applied to the sys- 
tems approach to artificial vision (see figure 21.3). 
(These technologies are also described in more de- 
tail in part II of this volume.) 



Technologies 



fMRI 



The Systems Perspective and 
the Role of New Technologies 

Given the above comments, it appears that deliver- 
ing simple patterns of electrical stimulation may 
not be adequate to create useful vision. This is not 
surprising given that visual information processing 
is a very complex process. Our approach to this 
complexity is systemic, including a very broad per- 
spective on the investigation of visual functions. 
Our approach incorporates not only engineering, 
information technology, single-cell physiology, 
ophthalmology, surgery, and material science, but 
also aims at improving our understanding of the 
mechanisms of encoding visual information to cre- 



Positron Emission Tomography (PET) and re- 
lated methods were the first techniques that 
yielded three-dimensional pictures of brain activity 
(Haxby Grady, Ungerleider, & Horwitz, 1991). 
However, those images provided only low spatial 
and temporal resolution, and the method was too 
invasive in that radioactive agents were required. 
When Magnetic Resonance Imaging (MRI) was 
modified to allow scanning of brain function 
(fMRI) in addition to anatomical imaging, neuro- 
science had found the low-risk method that could 
be applied to a wide range of populations and re- 
search questions and that yielded astonishingly de- 
tailed information on brain activity (figure 21.3A; 
see chapter 4, this volume, for more details on 
fMRI). 
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Figure 21.3. Techniques used to study brain function. (A) Example of a functional Magnetic Resonance Imaging 
scanner. (B) Magnetic encephalography Vectorview; Elekta Neuromag Oy Finland. (C) Transcranial Magnetic 
Stimulation. 



fMRI has yielded important information on 
mechanisms of visual processing (Wandell, 1999). 
In combination with other methods (e.g., EEG or 
MEG) with a higher temporal resolution, many 
behavior-based theories regarding the normal re- 
sponse pattern of the visual system have been con- 
firmed by this type of imaging. New studies have 
explored the observation of changes of brain activ- 
ity patterns over time, including over the relatively 
short time required for learning new tasks and over 
long periods of time to track neural development. 

EEG and MEG 

The EEG represents one of the earliest and sim- 
plest methods to noninvasively record human 
brain activity. By characterizing different patterns 
of brain wave activity, an EEG can identify a per- 
son's state of alertness and can measure the time 
it takes the brain to process various stimuli 
(Malmivuo, Suihko, & Eskola, 1997). While EEG 
has great temporal sensitivity, a major drawback 
of this technique is that it cannot reveal the 
anatomy of the brain nor identify with certainty 
the specific regions of the brain that are impli- 
cated with task performance. This issue, referred 
to as the inverse problem, defines that for a given 
recorded pattern of electrical activity, there are es- 
sentially an infinite number of configurations of 
current sources that could account for the 
recorded signal. To localize bioelectric sources 
within the brain more accurately, highly complex 
mathematical models and combined imaging 
techniques (e.g., fMRI) are used (Dale & Halgren, 
2001). EEG methods are described in more detail 
in chapter 2, this volume. 



Another related technology that provides greater 
temporal resolution of brain activity is MEG; figure 
21. 3B). MEG is a complex, noninvasive technique 
that attempts to detect, measure, record, and analyze 
the extracranial magnetic fields induced by the elec- 
trical activity of the brain. The skull is transparent to 
magnetic fields and the measurement sensitivity of 
MEG in the brain is theoretically more concentrated 
compared to EEG, since the skull has low conduc- 
tivity to electric current (Malmivuo et al., 1997). 
Over 100 magnetic detection coils (superconducting 
quantum interference devices or SQUIDs) are posi- 
tioned over the subject's head to detect bioelectrical 
activity. MEG has the advantage of not needing elec- 
trodes to be attached to the skin, and it provides 
both superior temporal and spatial resolution of 
measured brain activity. The raw data are filtered 
and processed by mathematical models to estimate 
the location, strength, and orientation of the current 
sources in the brain. Of all the brain-scanning meth- 
ods, MEG provides the most accurate resolution of 
the timing of neural activity, but this technique is ex- 
pensive to purchase, operate, and maintain (Lounas- 
maa, Hamalainen, Hari, & Salmelin, 1996). 

Lesion Studies 

To those not aware of basic visual system organiza- 
tion, it might seem odd that we see with the brain 
rather than with our eyes and that the part of the 
brain primarily receiving visual information is lo- 
cated at the back of the head, that is, at a maximal 
distance from the eyes. In fact, 27% of the cortex 
contributes in one way or another to visual percep- 
tion, and even more areas link visual functions to 
other perceptual modalities (Grill-Spector, 2003; 
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Figure 21 .4. Organization of sensory and motor areas shown on a three-dimensional volume-rendered brain 
(A) and flattened projection (B). For simplicity, only the right hemisphere is shown. Modified from van Essen 
(2004) with permission from MIT Press. See also color insert. 



Van Essen, 2004) that are distributed over even 
more extensive cortical and subcortical areas (see 
figure 21.4). Therefore, it is not surprising that any 
brain lesion bears a high risk of inducing some vi- 
sual impairment (Kasten et al., 1999). 

Most of the knowledge about the function of a 
particular brain area in humans is derived indi- 
rectly, for example, from brain imaging or electro- 
physiological studies (see below). However, such 
experiments usually cannot provide causal evidence 
that a specific function is supported by a specific 
brain area because there are many alternative expla- 
nations for any given data set (e.g., the interaction 
of several brain areas or inhibitory processes that 
modulate brain activity). Clear conclusions on 
causal connections between a brain function and its 
neuronal substrate have historically been obtained 
by lesion studies in which the effect of damaging a 
specific area of the brain on function is studied or 
by naturally occurring disease. The role of posterior 
parts of the brain in the generation of visual percep- 
tion has been slowly uncovered by way of "natural 
experiments" — that is, by observing correlations 
between specific brain lesions (most obvious in 
traumatic injury) and the loss of visual functions 
(Finger, 1994). Lesion studies in animals with ho- 
mologues of particular human brain regions, espe- 
cially primates, have provided information and 
confirmed hypotheses on the behavior of neuronal 
networks of vision in the brain. Remarkable single 
case studies, such as that of a patient with a selec- 
tive impairment of perceiving visual motion (Zihl, 



von Cramon, Mai, & Schmid, 1991), have shed 
light on the high degree of specialization of visual 
areas; that is, local brain damage affecting the mo- 
tion areas in extrastriate cortex results in a precisely 
defined loss of perceptual function, that is, percep- 
tion of visual motion. The global organization of the 
visual system in pathways for parallel processing of 
"where" and "what" information (Ungerleider & 
Haxby, 1994) was supported by predictable func- 
tional loss in patients with selective lesions to one 
of the pathways (Stasheff & Barton, 2001). 

The major drawback of this lesion approach is 
that many subjects have chronic lesions and infor- 
mation about the reorganization of the brain can- 
not be gained. Further, it is difficult to conclude 
that the findings from such patient studies actually 
reveal insight into the normal function of the visual 
brain in a healthy person. From the experimental 
standpoint, a disadvantage is that the lesions are 
not under control of the experimenter (e.g., the 
timing, size, and location). Better control can be 
achieved in animal studies, but structural homolo- 
gies between animals and humans do not necessar- 
ily imply functional similarity. Moreover, the 
introspection of patients, which is a useful tool of 
investigation in neuropsychology, cannot be ex- 
tracted from animals. New techniques make it pos- 
sible to temporarily block brain activity in specific 
locations, like, for instance, TMS, which permits 
the study of temporary and nonharmful lesion ef- 
fects in humans (see "Transcranial Magnetic Stimu- 
lation" below). As such, TMS enables performance 
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of nondestructive lesion studies in volunteers to 
learn more about brain function. 

Transcranial Magnetic Stimulation 

TMS is a noninvasive neurophysiological technique 
that can be used to transiently disrupt the function 
of the targeted brain area, functionally map cortical 
areas, assess the excitability of the stimulated cor- 
tex, and even modify its level of activity for a short 
period of time (figure 21. 3C). TMS is widely used 
to investigate complex aspects of human brain 
function such as memory, language, attention, 
learning, and motor function, and is even being 
studied as a potential therapy for depression 
(Kobayashi & Pascual-Leone, 2003; Merabet et al., 
2003; Walsh & Pascual-Leone, 2003). 

The physical basis of TMS rests on the princi- 
ples of electromagnetic induction originally dis- 
covered by Faraday in 1831. Simply stated, a brief 
electrical pulse of rapidly alternating current 
passed through a coil of wire generates a strong, 
transient magnetic field (on the order of 1 to 2 
Tesla). By placing a stimulating coil near a sub- 
ject's scalp, the induced magnetic field pulse pene- 
trates the scalp and skull virtually unattenuated 
(decaying only with distance) to reach the under- 
lying brain tissue. The rate of change of this mag- 
netic field induces a secondary electrical current 
in any nearby conducting medium, such as the 
brain tissue. Unlike direct electrical cortical stimu- 
lation, TMS does not require the underlying cor- 
tex to be exposed. In essence, TMS represents a 
method of electrical stimulation without elec- 
trodes, whereby the magnetic component is re- 
sponsible for bridging the transition between 
current passed in the coil and current induced in 
the brain. 

The effects of this induced current on brain tis- 
sue are dependent on numerous stimulation pa- 
rameters (such as stimulation frequency, intensity, 
and coil configuration and orientation). The volt- 
age of the primary current and the geometry of the 
stimulating coil determine the strength and shape 
of the magnetic field and thus the density and spa- 
tial resolution of the secondary current induced in 
the tissue. There are two main coil designs: (1) cir- 
cular coils, which generate large currents and are 
useful for stimulating relatively large cortical areas; 
and (2) figure eight coils, which are more focal 
(owing to a maximal current being generated at the 



intersection of the two round coil wings). Current 
evidence suggests that the locus of TMS stimulation 
has a spatial resolution of 1 cm 2 with a penetration 
depth of approximately 2 cm using commercially 
available, figure eight stimulation coils. To stimu- 
late at greater depths, larger amounts of current are 
required, but this comes at the expense of spatial 
resolution. 

Typically, a single pulse lasts 200 ms; however, 
stimulation can also be delivered as a train of repet- 
itive pulses (referred to as repetitive TMS or rTMS). 
Unlike the single pulse, repetitive TMS can 
modulate cortical excitability with effects lasting far 
beyond the period of stimulation itself (Pascual- 
Leone et al., 1998). While significant interindi- 
vidual differences in responsiveness exist, it is 
generally accepted that motor excitability can be 
enhanced by stimulating at high frequencies (i.e., 
5-25 Hz). Conversely, low frequency rTMS (at 
1 Hz) lowers cortical excitability and is thought to 
result in cortical inhibition (Pascual-Leone et al., 
1998). 

From a safety standpoint, the great concern of 
using TMS is the risk of inducing a seizure (particu- 
larly using repetitive TMS and in subjects with a 
known prior history of epilepsy or a predisposing 
factor). Since the institution of safety guidelines 
defining stimulation parameters and the monitoring 
and screening of appropriate subjects, no TMS- 
induced seizures have been reported (Wassermann, 
1998). 

TMS has also been used extensively in the 
mapping of other, nonmotor cortical targets in- 
cluding frontal, temporal, and occipital areas in re- 
lation to studies involving memory, attention, 
language, visual perception, and mental imagery. A 
TMS pulse delivered with appropriate parameters 
can temporally disrupt cortical function in a given 
area. More recently, TMS has been used to supple- 
ment and refine conclusions drawn from neu- 
roimaging studies (such as PET and fMRI). By 
establishing a causal link between brain activity 
and behavior, TMS can be used to identify which 
parts of a given network are necessary for perform- 
ing a behavioral task. For example, when a TMS 
pulse is delivered at a specific time to the visual 
cortex of the brain, the detection of a presented vi- 
sual stimulus can be blocked. Using this paradigm, 
the timing between the presentation of a visual 
stimulus and TMS pulse has been determined by 
varying the chronometry of visual perception 
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within primary and extrastriate visual cortical areas 
(Amassianet al., 1998). 

As mentioned earlier, TMS can be used to es- 
tablish functional significance of data obtained 
through neuroimaging. For example, by combining 
PET imaging and TMS, the common neurophysio- 
logical substrate between visual perception and 
mental visual imagery can be investigated. Data 
collected from PET studies have suggested that 
similar visual areas are activated whether patients 
mentally imagine a visual scene or view it directly. 
TMS delivered to primary visual cortex disrupts 
not only visual perception but visual imagery, 
which suggests that occipital cortex is necessary in 
both these processes (Kosslyn et al, 1999). 

The Top-Down or Systems Approach 

In what way does the recent development of brain 
imaging and stimulation methods described above 
change the approach of creating a retinal prosthe- 
sis? To understand how the visual system works, 
the higher stages of processing beyond the level of 
the retina and the eye have to be considered, be- 
cause many crucial aspects of the perceptual pro- 
cess are shaped at the thalamus and various 
cortical locations. Consideration of all stages of 
processing along the visual pathway is essential to 
develop a retinal prosthesis, which must in some 
way talk to the brain. Such "smart" devices must be 
developed with this knowledge about brain pro- 
cesses, because the visual cortex is not only a pas- 
sive receptor of visual information from lower 
stages of processing, but — guided by influences 
from higher-level visual and cognitive areas of the 
brain — it actively changes the input signals. For 
example, what we see is not just determined by the 
pattern of photons or different wavelengths of light 
striking the retina — the final percept depends on 
previous experience, expectation, current inten- 
tions, and motivation, as well as attentional and 
other inner states. Thus, a wide range of brain ar- 
eas contribute to the perception of sight. As has 
been described above, early experiments with reti- 
nal implants in our group (Rizzo et al., 2003a, 
2003b) showed that electrical stimulation of the 
inner retina induces percepts, but that a score- 
board approach of conveying stimulation patterns 
to higher visual areas was not tenable. Similar re- 
sults were later found in patients who had been 
chronically implanted with a retinal prosthesis 



(Humayun et al., 2003). For creating a smart stim- 
ulation device, we first have to understand the rela- 
tionship of lower-level signals with high-level brain 
activity during visual perception. But we must also 
learn more about the interaction between bottom- 
up signals from the retina and top-down signals 
from frontal and parietal areas of the brain and 
higher areas of the visual pathway. 

As a first step, we should also consider that 
blindness changes the brain and the way percep- 
tual information is processed by the remaining, in- 
tact parts of the visual system (Grusser & Landis, 
1991). The brain and the sequences of signal pro- 
cessing are not static, but flexibly adapt to transient 
alterations of internal or external aspects of percep- 
tion and to long-term changes such as occur with 
lesions of the visual pathway that create blindness. 
The techniques of brain imaging and stimulation 
described above are useful for investigating such 
processes of brain plasticity and provide relevant 
information that will influence the functional de- 
sign of our retinal prostheses. Moreover, with the 
assistance of fMRI, TMS, and other imaging and 
stimulation methods, we should also be able to 
learn more about strategies to develop a retinal 
prostheses and its effects on the activity of the vi- 
sual system and the brain in general. 



Brain Plasticity 

Visual System Plasticity and Influence 
of Cognitive Factors 

The earliest neuropsychological theories about 
brain function were strongly influenced by the idea 
that any given capacity was represented in a local- 
ized brain area and that each region was assigned a 
specific function that was immutable. More specifi- 
cally, by the 1940s and 1950s, the highly specific 
processing of information and the strict topograph- 
ical organization of the visual system had been 
discovered (see Grusser & Landis, 1991). These 
discoveries advanced the belief that brain plasticity 
and functional recovery are impossible, or at least 
extremely limited. Until recently, the visual system 
was mainly regarded as hard-wired. The potential 
of the visual brain to heal itself was not appreci- 
ated (see Pambakian & Kennard, 1997; Sabel, 
1999), other than within a critical period early in 
life. Even then, selective deprivation of certain 
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aspects of visual experience in young animals was 
shown to produce a massive effect on the architec- 
ture of the visual system that led to permanent 
functional impairment (Sabel, 1999; Wiesel & 
Hubel, 1965). 

However, more recent animal experiments as 
well as human research have shown that visual 
deficits induced by lesions or deprivation can ei- 
ther recover spontaneously or that function can be 
regained by systematic training even beyond the 
critical period (Daas, 1997; Freund et al., 1997; 
Kasten et al., 1999; Poggel, Kasten, Miiller-Oehring, 
Sabel, & Brandt, 2001; Sabel, 1999). These newer 
findings have documented a previously unappreci- 
ated plasticity of the visual system that can provide 
improved function, albeit the exact mechanisms of 
reorganization are not yet clear. Notwithstanding 
this potential for plasticity, in a recent debate, the 
outcomes from human training trials are criticized 
as not being relevant to the everyday life of patients 
and possibly being due to artifacts (e.g., a shift of 
the patient's fixation toward the blind field; see 
Horton, 2005; Plant, 2005, for a critical review). 
Obviously, more work must be performed in this 
emerging area to better define the capacity of the 
visual system to recover after injury, and to judge 
how this capacity is affected by age, and the extent 
and location of a lesion. 

Earlier behavioral and electrophysiological ani- 
mal experiments indicated that retinal and cortical 
lesions are followed by widespread reorganization 
of receptive field structures (Chino, Smith, Kaas, 
Sasaki, & Cheng, 1995; Eysel et al, 1999). In these 
experiments, cortical activity was initially sup- 
pressed within cortical areas that had previously 
received input from a lesioned area of retina or the 
border zones of cortical scotoma. Even minutes af- 
ter a lesion of the retina or cortex, the receptive 
fields of neurons adjacent to a lesioned area in- 
creased considerably in size (up to three times the 
original size) and expanded into the region origi- 
nally represented by the lesioned part (Eysel et al, 
1999; Gilbert, 1998; Kasten et al., 1999; Sabel, 
1997, 1999). Hence, the representation of the le- 
sioned area in the cortex shrinks over time, and 
neural activation recovers. This process of recovery 
seems to be triggered by the areas surrounding the 
lesion, which show increased spontaneous activa- 
tion and excitability by visual stimulation. In addi- 
tion to the quick increase of receptive field size 
immediately after the lesion, there are also slower 



processes of receptive field plasticity that take place 
over weeks to months. Thus, retinal and cortical 
defects of the visual system have immense effects 
on the topographical maps in the visual cortex 
(see Eysel et al., 1999, for a review). These pro- 
cesses take place in the mature visual system 
and may be based on molecular and morphologi- 
cal mechanisms similar to those that have been 
found to occur during the normal process of brain 
development. 

In the beginning of the 20th century, the first 
systematic observations of spontaneous recovery of 
visual functions in humans were made (e.g., Pop- 
pelreuter, 1917, cited in Poggel, 2002), mainly in 
soldiers surviving brain lesions during the two 
world wars. Since then, many investigators have 
contributed findings in favor of some degree of 
spontaneous recovery in patients with visual field 
loss. However, the results concerning the duration 
of the period of recovery, the size of the visual field 
regained, and predictors of recovery were often 
contradictory (see Poggel, 2002, for an overview). 
Still, the observation that vision can improve even 
months after a lesion of the visual system prompts 
the question as to whether processes of recovery 
can be actively manipulated to enhance the process 
of visual rehabilitation. Notwithstanding these in- 
sights, for several decades, treatment approaches 
for patients with partial blindness concentrated on 
the compensation for the visual field defect and did 
not aim at a restoration of the lost function (Kasten 
et al., 1999; Kerkhoff, 1999). 

In the 1960s and 1970s, training-induced re- 
covery of visual function was first shown in animals 
(Cowey, 1967; Mohler & Wurtz, 1977), which was 
followed by attempts at restoring visual function in 
patients (Zihl & von Cramon, 1979, 1985). In the 
meantime, many studies supported the initial evi- 
dence of vision restoration (see Kasten et al., 1999, 
for an overview). The benefit of training has been 
asserted for patients with optic nerve lesions and 
postgeniculate lesions in one prospective, random- 
ized, placebo-controlled clinical trial (Kasten, Wust, 
Behrens-Baumann, & Sabel, 1998). In this study, 
patients who were trained with a computer-based 
program that stimulated the border regions of the 
defective visual area showed a larger visual field and 
a higher number of detected light stimuli in high- 
resolution visual field testing after 6 months of 
training, versus no change in a control group who 
had performed a placebo fixation training. Then, 
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Poggel, Kasten, Muller-Oehring, Sabel, and Brandt 
(2001) compared processes of spontaneous and 
training-induced recovery of vision in a patient 
with a shotgun lesion. The phenomenology and 
topography of training-induced and spontaneous 
recovery were very similar, which supports a hy- 
pothesis that similar mechanisms may underlie 
both processes of recovery 

The amount of residual vision at the border of 
the blinded area of the visual field is the main pre- 
dictor of training success (Kasten et al., 1998; 
Poggel, Kasten, & Sabel, 2004). Presumably, par- 
tially defective brain regions, at the border of the 
lesion, represent cortical areas where vision is not 
completely lost but is quite impaired. In border ar- 
eas, perimetric thresholds are increased, discrimi- 
nation of forms and colors is impossible, reaction 
times are prolonged, and the subjective quality of 
perception is reduced. However, systematic stimu- 
lation of these areas of residual vision seems to be 
able to reactivate the partially defective regions, 
possibly by co-activating neurons connected to the 
defective regions via long-range horizontal connec- 
tions in the cortex. These factors may increase re- 
ceptive field size and reduce the size of the blind 
field (Kasten et al., 1999; Poggel, 2002). 

In a more recent study, an attentional cue was 
used to help the patients focus attention at the vi- 
sual field border, specifically to the areas of resid- 
ual vision that are crucial for regaining visual 
functions. This strategy increased the training ef- 
fect in patients who received the cue versus the re- 
sults obtained from a group of participants who 
performed the conventional visual field training 
without attentional cueing (Poggel, 2002; Poggel et 
al., 2004). This beneficial effect of the cue might be 
due to a systematic combination of bottom-up 
stimulation (i.e., the light stimuli presented during 
the training) with top-down attentional activation. 

As alluded to above, the effects of visual field 
training are highly controversial (Horton, 2005; 
Pambakian &r Kennard, 1997; Plant, 2005; see also 
Sabel, 2006). While it remains to be proven in fu- 
ture studies that the treatment effects are clinically 
relevant and cannot be explained by artifacts, there 
appears to be growing evidence from animal and 
human studies of a heretofore unrecognized poten- 
tial for cortical plasticity. 

The example of the study by Poggel et al. 
(2004) on attention effects in visual field training 
shows that more than just the neural output from 



the retina should be taken into account when ther- 
apeutic measures are planned. Top-down influ- 
ences may be systematically applied to improve 
visual performance in patients. Cognitive aspects 
of vision also have to be included to understand 
the various effects that bottom-up stimulation can 
have on perception. A percept cannot be predicted 
solely on the basis of the bottom-up signal 
(Loewenstein et al., 2004; Rizzo et al., 2003a, 
2003b); factoring in the effects of top-down signals 
may make the effects of stimulation, such as those 
induced by a retinal prosthesis, more predictable. 
The situation becomes even more complex because 
the visual cortex cannot be looked at in isolation, 
given that cortical plasticity takes place in many 
nonvisual areas that interact with the visual system. 

Cross-Modal Plasticity 

Neuroplasticity in the Blind and Cross-Modal 
Sensory Integration 

The existence of specialized receptors for different 
sensory modalities provides the opportunity to 
process different forms of input and hence capture 
different views of the world in parallel. While some 
experiences are uniquely unimodal (e.g., color and 
tone), the different sensory systems presumably 
closely collaborate with each other, given that our 
perceptual experience of the world is richly multi- 
modal and seamlessly integrated (Stein & Mered- 
ith, 1990). Despite the overwhelming evidence that 
we integrate multimodal sensory information to ob- 
tain the most accurate representation of our envi- 
ronment, our thinking about the brain is shaped by 
historical notions of parallel systems specialized for 
different sensory modalities. For each of those 
modalities, we assume a hierarchically organized 
system that begins with specialized receptors that 
feed unimodal primary cortical areas. A series of 
secondary areas unimodally integrate different as- 
pects of the processed information. Eventually, mul- 
timodal association areas integrate the processed 
signals with information derived from other sensory 
modalities. 

But what happens when parts of the brain that 
process a given modality of sensory information 
are separated from their input? It seems reasonable 
to presume that the brain would reorganize, so that 
neuroplasticity would compensate for the loss in 
concert with the remaining senses. The study of 
blind individuals provides insight into such brain 
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reorganization and behavioral compensations that 
occur following sensory deprivation. For instance, 
Braille reading in the blind is associated with a va- 
riety of neuroplastic changes, including develop- 
ment of an expanded sensorimotor representation 
of the reading finger (Pascual-Leone & Torres, 
1993; Pascual-Leone et al., 1993) and recruitment 
of the occipital cortex during tactile discrimination 
(Buchel, Price, Frackowiak, & Friston, 1998; Bur- 
ton et al, 2002; Sadato et al., 1996, 1998). These 
findings suggest that pathways for tactile discrimi- 
nation change following blindness. Indeed, 
somatosensory-to-visual cross-modal plasticity has 
been shown to be behaviorally relevant, given that 
interfering with the visual cortex using TMS can 
disrupt Braille reading in blind patients (Cohen 
et al., 1997). Converging evidence suggests that 
cross-modal plasticity following sensory depriva- 
tion might be a general attribute of the cerebral 
cortex. For instance, Weeks et al. (2000) have 
shown recruitment of occipital areas in congeni- 
tally blind individuals during a task that required 
auditory localization. Interestingly, the opposite re- 
lation also holds true. Activation of the auditory 
cortex was observed in a sample of profoundly deaf 
subjects in response to purely visual stimuli 
(Finney, Fine, & Dobkins, 2001) and vibrotactile 
stimulation using MEG (Levanen, Jousmaki, & 
Hari, 1998). These reports can be interpreted as 
meaning that activation of the occipital cortex dur- 
ing processing of nonvisual information may not 
necessarily represent the establishment of new 
connections, but rather the unmasking of latent 
pathways that participate in the multisensory per- 
ception. This functional recruitment of cortical ar- 
eas may be achieved by way of rapid neuroplastic 
changes. These observations support the belief that 
unimodal sensory areas are part of a network of ar- 
eas subserving multisensory integration and are 
not merely processors of a single modality. 

In parallel with development of visual prosthe- 
ses, great strides are being made in other areas of 
neuroprosthetic research. Cochlear implant re- 
search has enjoyed the most progress and success 
to date (see Loeb, 1990) and in many respects has 
served as an impetus and model for visual pros- 
thetic development. Deaf individuals learn to use 
cochlear implants by establishing new associations 
between sounds generated by the device and ob- 
jects in the auditory world. However, there remains 
great intersubject variability in adapting to 



cochlear implants, with speech recognition perfor- 
mance ranging from very poor to near perfect 
among various patients. 

In many cases, specific rehabilitation strategies 
for prospective recipients of a cochlear implant 
have to be tailored to the profile of the candidate in 
order to maximize the likelihood of success. For 
instance, the assessment of the degree of visual- 
auditory cross-modal plasticity in the deaf has also 
been extended to patients receiving cochlear im- 
plants. Using PET imaging, Lee et al. (2001) found 
that in prelingually deaf patients (i.e., hearing loss 
prior to the development of language), the primary 
auditory cortex was activated by the sound of spo- 
ken words following receipt of a cochlear implant 
device. Even more astounding, this group found 
that the degree of resting hypometabolism before 
device implantation was positively correlated with 
the amount of improvement in hearing capability 
after the operation. These authors suggest that if 
cross-modal plasticity restores metabolism in the 
auditory cortex before implantation, the auditory 
cortex may no longer be as able to respond to sig- 
nals from a cochlear implant, and prelingually deaf 
patients will show no improvement in hearing 
function (despite concentrated rehabilitation) after 
implantation of a cochlear prosthesis. 

The results of these studies have important im- 
plications. First, they suggest that functional neu- 
roimaging may have a prognostic value in selecting 
those individuals who are ideal candidates for a 
cochlear implant. It is conceivable that a corollary 
scenario exists regarding cross-modal sensory inter- 
actions within the visual cortex of the blind, and we 
therefore suspect that such a combined approach 
(i.e., using information obtained by neuroimaging) 
may help identify candidates who are most likely to 
succeed with a visual prosthesis implant. Put an- 
other way, simple reintroduction of lost sensory in- 
put may not suffice to restore the loss of a sense. We 
hypothesize that specific strategies will be needed 
to modulate brain processing and to enhance the 
extraction of relevant and functionally meaningful 
information generated from neuroprosthetic inputs. 

Is it possible to exploit a blind person's existing 
senses in order to learn how to see again? Tapping 
into such mechanisms may be advantageous for 
enhancing the integration of the encoding of mean- 
ingful percepts by a prosthesis. Clearly, there exists 
a correspondence between how an object appears, 
how it sounds, and how that same object feels 
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Figure 21 .5. The multimodal nature of our sensory world and its implications 
for implementing a visual prothesis to restore vision. (A) Under normal condi- 
tions, the occipital cortex receives predominantly visual inputs but also from 
cross-modal sensory areas. (B) Following visual deprivation, neuroplastic changes 
occur such that the visual cortex is recruited to process sensory information from 
other senses (illustrated by larger arrows for touch and hearing). (C) After neuro- 
plastic changes associated with vision loss have occurred, the visual cortex is fun- 
damentally altered in terms of its sensory processing, so that simple 
reintroduction of visual input (by a visual prosthesis; dark gray arrow) is not suf- 
ficient to create meaningful vision (in this example, a pattern encoding a moving 
diamond figure is generated with the prosthesis). (D) To create meaningful visual 
percepts, a patient who has received an implanted visual prosthesis can incorpo- 
rate concordant information from remaining sensory sources. In this case, the di- 
rectionality of a moving visual stimulus can be presented with an appropriately 
timed directional auditory input, and the shape of the object can be determined 
by simultaneous haptic exploration. In summary, modification of visual input by 
a visual neuroprosthesis in conjunction with appropriate auditory and tactile 
stimulation could potentially maximize the functional significance of restored 
light perceptions and allow blind individuals to regain behaviorally relevant vi- 
sion. From Merabet et al. (2005). Reproduced with permission from Nature Re- 
views Neuroscience , copyright 2005 Macmillan Magazines Ltd.). 



when explored through touch. Not surprisingly, 
functional neuroimaging has demonstrated a signif- 
icant overlap between cortical areas involved in ob- 
ject recognition through sight and touch (e.g., 
Amedi, Malach, Hendler, Peled, & Zohary, 2001, 
2002; Beauchamp, Lee, Argall, & Martin, 2004; 
Diebert, Kraut, Kremen, & Hartt, 1999; James 
et al, 2002; Pietrini et al., 2004). It thus appears 
that visual cortical areas — once considered part of a 



specialized system — are involved in multiple forms 
of sensory processing, and they incorporate widely 
distributed and overlapping object representations 
in their processing schemes. This notion implies 
that a crucial aspect of information processing in 
the brain is not solely dependent upon a specific 
kind of sensory input, but rather on an integrated 
computational assimilation across diverse areas of 
the brain (Pascual-Leone & Hamilton, 2001). We 
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hypothesize that the tactile and auditory inputs 
(processed in the cross-modally changed brain) can 
be used to remap restored visual sensations and, in 
turn, help devise and refine visual stimulation 
strategies using a common conceptual sensory 
framework (for discussion, see Merabet et al., 2005; 
see figure 21.5). Given that sensory representations 
are shared, appropriate tactile and auditory inputs 
can assist a patient in using a visual neuroprosthesis 
to functionally integrate concordant sources of sen- 
sory stimuli into meaningful percepts. 



Conclusion 

This chapter has reviewed broad neuroscience top- 
ics and new technologies that may enhance the un- 
derstanding of how to create useful vision for blind 
persons and for related neuroergonomic applica- 
tions. This discussion has emphasized the fact that 
the visual system does not operate in isolation. 
What a patient will see with a retinal implant will 
relate to the processes of plasticity and perhaps 
other perceptual modalities, as well as top-down 
influences like attention that play a significant role 
in shaping the percept. Using that information and 
the new technologies available for visual neuro- 
science, it may be possible to develop smarter 
devices and to make visual rehabilitation more effi- 
cient. 

Creating a retinal implant is a challenge for 
ophthalmologists and surgeons, electrical engi- 
neers, and material scientists. But the creation of 
retinal implants will also have an impact on neuro- 
science disciplines. Once the mechanisms of visual 
stimulation and processing incoming signals are 
better understood, another demanding task will be 
the development of smart algorithms to control the 
neuroprosthesis and to implement learning proce- 
dures that will allow tuning of the implant to the 
patient's needs. This process will require close co- 
operation with cognitive scientists, perceptual spe- 
cialists, and the patients who will hopefully benefit 
from these efforts. 



2. Retinal prostheses may help to restore useful 
visual performance and improve the quality of 
life in blind patients. 

3. Efforts to create visual prosthetics have been 
partially successful but are limited mainly by 
our inability to communicate with the brain. 

4. A systems perspective, taking into account 
state-of-the-art neuroimaging methods and 
results from cognitive research and 
neuropsychology, may help to create "smart" 
visual prosthetics. 

5. Visual brain plasticity and the interaction of 
the visual system with other sensory 
modalities have to be included in the effort to 
create a retinal implant that is able to talk to 
the brain. 

6. Advances in microtechnology and the 
possibility of creating artificial vision create 
new challenges for the field of rehabilitation. 
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The anatomical structure and function of the hu- 
man motor system is highly complex (figure 22.1). 
Billions of central nervous system (CNS) neurons 
in the cerebral cortex, cerebellum, brain stem, and 
spinal cord are involved in planning and executing 
movements. These efferent motor signals propagate 
via millions of upper motor neurons through the 
spinal cord to the lower motor neurons of the pe- 
ripheral nervous system (PNS) that innervate the 
skeletal muscles. Muscle contractions take place in 
thousands of independent motor units. Spatial and 
temporal modulation of motor signals adjusts mus- 
cle forces to produce smooth movements. Dozens 
of muscles span single or multiple skeletal joints to 
generate specific limb movements or maintain cer- 
tain body postures. Millions of receptors in joints, 
muscles, tendons, and the skin sense muscle forces 
and movements and send this feedback informa- 
tion via afferent (sensory) nerve fibers in the PNS 
back to the CNS. This information is processed 
and compared with the planned movement, to- 
gether with visual inputs and inertial (vestibular) 
inputs on the position of the limbs and body in 
space to ensure accuracy. In general, the brain gen- 
erates and controls voluntary movements, whereas 
central pattern generators in the spinal cord medi- 
ate a variety of stereotypical movements. These 



movements can be affected by lesions of the CNS, 
PNS, and the musculoskeletal system (table 22.1). 
As we shall see, various disorders caused by lesions 
along these pathways can be mitigated with a vari- 
ety of new strategies and devices. 



Pathologies of the Human 
Motor System 

As mentioned, human movement relies on a vari- 
ety of neural pathways between the brain, spinal 
cord, and sensory (afferent) receptors. These le- 
sions can be caused by tumor, trauma, strokes 
(both hemorrhagic and ischemic), tumors, infec- 
tions, demyelinative disorders, systemic disorders, 
and a host of other conditions that are reviewed in 
standard textbooks of neurology and internal med- 
icine. Briefly, focal brain lesions in the primary mo- 
tor cortex or other CNS regions responsible for the 
planning and generation of motor behavior may 
impair limb movements on the opposite side of the 
body (known as a hemiparesis when there is some 
residual function or hemiplegia when there is total 
paralysis); the opposite arm or leg may be affected 
alone (known as a monoparesis or monoplegia), 
depending on the size and location of the lesions. 
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Figure 22.1 . Main anatomical components of the human motor system. Adapted from 
Shumway-Cook & Woollacott (2001). 



Table 22.1. Relation Between Injured Region, Physiological Function, Pathology, and Methods of Restoration 



Region of Lesion 



Functions Involved 



Possible Pathologies 



Possible Methods of Restoration 



Brain 



Spinal cord 



Motion planning 
Stimulus generation 



Stimulus generation 
Stimulus propagation 

Peripheral nervous Stimulus propagation 

system 



Muscles 



Skeletal system 



Motion execution 
Body posture 



Motion execution 
Body posture 



Stroke, trauma, tumor, 

demyelinating disease 

(e.g, multiple sclerosis, 

Parkinson disease, cerebral 

palsy) 

Spinal cord injury 

(paraparesis, tetraparesis) 

Neuropathy, neurapraxia, 

after peripheral nerve 

injuries 

Myopathy, trauma 



Osteoporosis, arthrosis 

Fracture 

Amputation 



Surgical intervention 
Motion therapy 
Neuroprosthesis 
Orthosis 

Neuroprosthesis 

Orthosis 

Spontaneous healing 

Surgical intervention 

Orthosis 

Spontaneous healing 

Orthosis 

Robotic support 

Exoprosthesis 

Spontaneous healing 

Orthosis 

Robotic support 

Exoprosthesis 
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Lesions of the cerebellum can affect the execu- 
tion of fine motor tasks and goal-directed motor 
tasks without producing weakness. This can be 
characterized by poorly synchronized activities of 
muscle agonists and antagonists (asynergia) lead- 
ing to (intention) tremor during movement, unco- 
ordinated limb movements (appendicular ataxia), 
and unsteady posture and gait (truncal ataxia). 

Spinal cord injuries (SCI) interrupt signal trans- 
fer between the brain and the periphery (muscles, 
receptors), causing loss of sensory and motor func- 
tions. Depending on lesion location and size, this 
causes partial or complete loss of tactile and kines- 
thetic sensations, and loss of control over voluntary 
motor functions, respiration, bladder, bowel, and 
sexual functions. Lumbar and thoracic spinal cord 
lesions impair lower extremity sensation and move- 
ment (paraparesis, paraplegia), whereas cervical 
cord lesions affect lower and upper extremities and 
the trunk (tetraparesis, tetraplegia). Lesions above 
the third cervical vertebra (C3) can lead to a loss of 
breathing function and head movements. 

Lesions produce spasticity when upper motor 
neurons are damaged and lower motor neurons re- 
main intact. Patients with spasticity show pathologi- 
cally brisk tendon reflexes and an increased muscle 
tone, which can lead to contractures of a limb. These 
patients may be treated with electrical stimulation. In 
contrast, lesions produce atonia (or hypotonia) when 
lower motor neurons are injured. Compared to le- 
sions that cause spasticity, lesions that produce ato- 
nia (hypotonia) reduce reflex activity and muscle 
tone and produce more striking muscle atrophy. 

Finally, musculoskeletal function can be affected 
by muscle diseases (myopathies), bone diseases 
(such as osteoporosis), and by tumors and trauma. 
Musculoskeletal injuries include muscle fiber rup- 
tures, ligament ruptures, joint cartilage damage, 
and bone fractures. Degenerative joint lesions (os- 
teoarthritis) can result from chronic posture imbal- 
ances and inappropriate joint stress. Even entire 
body limbs can be lost after accidents or surgically 
removed because of tumors or metabolic diseases 
(e.g., diabetes). Clearly, these conditions offer many 
challenges and novel opportunities for treatment. 



artificial mechanisms (table 22.1). Briefly, the body 
may use three natural mechanisms to restore func- 
tions: (1) Areas not affected by the lesion may par- 
tially compensate for lost functions; (2) functions 
of injured brain regions may be transferred to non- 
affected brain regions by generation of new synap- 
tic connections due to CNS plasticity; and (3) 
damaged brain regions may regenerate to some de- 
gree. These mechanisms may be enhanced and ac- 
celerated by pharmaceutical, physiotherapeutic, or 
surgical treatments. 

Minor peripheral nerve damage (neuroprax- 
ies) can heal without any additional treatment. Af- 
ter full nerve transection, an artificial nerve graft 
can be surgically inserted to support nerve 
growth. Natural restoration of the musculoskeletal 
system is limited to healing effects of muscles and 
bones, for example, after muscle fiber lesions or 
bone fracture. 

If the impairment of the nervous or muscu- 
loskeletal system cannot be restored by natural 
mechanisms, artificial technical support is re- 
quired. Totally lost functions can be substituted by 
prostheses, whereas orthoses are used to support 
remaining (but impaired) body functions. Exam- 
ples of mechanical prostheses are artificial limbs 
(exoprostheses) or artificial joints (endoprosthe- 
ses). Lesions in the CNS can be substituted by neu- 
roprostheses, which generate artificial stimuli in 
the PNS by functional electrical stimulation. A me- 
chanical orthosis is an orthopedic apparatus used 
to stabilize, support, and guide body limbs during 
movements. Typical examples for mechanical or- 
thoses are crutches, shells, gait and stance orthoses, 
and wheelchairs. 

In the following sections, two examples of 
movement restoration principles are described in 
more detail. In the first example, it is shown how 
natural restoration principles of the CNS can be 
enhanced by robot-aided motion therapy. In the 
second example, principles and problems of neu- 
roprostheses are clarified. 



Neurorehabilitation Robotics 



Natural and Artificial Mechanisms 
of Movement Restoration 

Motion impairments due to neural and muscu- 
loskeletal lesions can be restored by natural and 



Rationale for Movement Therapy 

Task-oriented repetitive movements can improve 
muscular strength and movement coordination in 
patients with impairments due to neurological or or- 
thopedic problems. A typical repetitive movement is 
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the human gait. Treadmill training has been shown 
to improve gait and lower limb motor function in 
patients with locomotor disorders. Manually as- 
sisted treadmill training was first used in the 1990s 
as a regular therapy for patients with SCI or stroke. 
Currently, treadmill training is well established at 
most large neurorehabilitation centers, and its use 
is steadily increasing. Numerous clinical studies 
support the effectiveness of the training, particu- 
larly in SCI and stroke patients (Barbeau & Rossig- 
nol, 1994; Dietz, Colombo, & Jensen, 1994; Hesse 
etal., 1995). 

Similarly, arm therapy is used for patients 
with paralyzed upper extremities after stroke or 
SCI. Several studies prove that arm therapy has 
positive effects on the rehabilitation progress of 
stroke patients (see Platz, 2003, for review). Be- 
sides recovering motor function and improving 
movement coordination, arm therapy serves also 
to teach new motion strategies, so-called trick 
movements to cope with activities of daily living 
(ADL). 

Lower and upper extremity movement therapy 
also serves to prevent secondary complications 
such as muscle atrophy, osteoporosis, and spastic- 
ity. It was observed that longer training sessions 
and a longer total training duration have a positive 
effect on the motor function. In a meta-analysis 
comprising nine controlled studies with 1,051 
stroke patients, Kwakkel, Wagenaar, Koelman, 
Lankhorst, and Koetsier (1997) showed that in- 
creased training intensity yielded positive effects 
on neuromuscular function and ADLs. This study 
did not distinguish between upper and lower ex- 
tremities. The finding that rehabilitation progress 
depends on training intensity motivates the appli- 
cation of robot-aided arm therapy. 

Rationale for Robot-Aided Training 

Manually assisted movement training has several 
major limitations. The training is labor intensive, 
and therefore training duration is usually limited 
by personnel shortage and fatigue of the therapist, 
not by that of the patient. During treadmill train- 
ing, therapists often suffer from back pain be- 
cause the training has to be performed in an 
ergonomically unfavorable posture. The disad- 
vantageous consequence is that the training 
sessions are shorter than required to gain an opti- 
mal therapeutic outcome. Finally, manually as- 
sisted movement training lacks repeatability and 



objective measures of patient performance and 
progress. 

In contrast, with automated (i.e., robot-assisted) 
gait and arm training, the duration and number of 
training sessions can be increased, while reducing 
the number of therapists required per patient. 
Long-term automated therapy can be an efficient 
way to make intensive movement training afford- 
able for clinical use. One therapist may be able to 
train two or more patients in the future. Thus, per- 
sonnel costs can be significantly reduced. Further- 
more, the robot provides quantitative measures, 
thus allowing the observation and evaluation of the 
rehabilitation process. 



Automated Gait-Training Devices 

One commercially available system for locomotion 
therapy is the Gait Trainer from the German com- 
pany Reha-Stim in Berlin (Hesse & Uhlenbrock, 
2000). Here, the feet of the patient are mounted on 
two separate plates that move along a trajectory 
that is similar to a walking trajectory. The device 
does not control knee and hip joints. Thus, the pa- 
tient still needs continuous assistance by at least 
one therapist. Forces or torques can only be mea- 
sured in the moving plates and not in the leg 
joints. 

Reinkensmeyer, Wynne, and Harkema (2002) 
developed a different automated gait trainer, which 
is characterized by several separate actuator units 
that are spanned between a static frame and the pa- 
tient. 

A third device is the Lokomat (Colombo, Jbrg, 
Schreier, & Dietz, 2000; Colombo, Jorg, & Jez- 
ernik, 2002), which is a bilateral robotic orthosis 
being used in conjunction with a body weight sup- 
port system to control patient leg movements in 
the sagittal plane (figure 22.2). The Lokomats hip 
and knee joints are actuated by linear drives, which 
are integrated in an exoskeletal structure. A passive 
elastic foot lifter induces an ankle dorsiflexion dur- 
ing the swing phase. The legs of the patient are 
moved with highly repeatable predefined hip and 
knee joint trajectories on the basis of a position 
control strategy. Knee and hip joint torques can be 
measured via force sensors integrated inside the 
Lokomat (Jezernik, Colombo, & Morari, 2004). 
The Lokomat is currently being used in more than 
30 different clinics and institutes around the 
world. 
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Figure 22.2. Current version of the 
Lokomat (Hocoma AG) . 




Automated Training Devices 
for the Upper Extremities 

Hesse, Schulte-Tigges, Konrad, Bardeleben, and 
Werner (2003) developed an arm trainer for the 
therapy of wrist and elbow movements. Each hand 
grasps a handle and can be moved in one degree of 
freedom (DOF). The device position has to be 
changed depending on the selected movement. 
Force and position sensors are used to enable dif- 
ferent control modes, including position and im- 
pedance control strategies. 

Another one-DOF device is the arm robot from 
Cozens (1999), which acts like an exoskeleton for 
the elbow joint. Interactive assistance is provided 
on the basis of position and acceleration signals 
measured by an electrogoniometer and an ac- 
celerometer. 

The Haptic Master is a three-DOF robot de- 
signed as haptic display by Fokker Control Sys- 
tems, FCS (Van der Linde, Lammertse, 
Frederiksen, & Ruiter, 2002). It has formed the 
basis of the GENTLE/s project supported by the 
European Union (Harwin et al., 2001). In this 
project, it was suggested to use the Haptic Master 
as a rehabilitation device for the training of arm 



movements by attaching the wrist of the patient to 
the end-effector of the robot. However, this setup 
yields an undetermined spatial position for the el- 
bow. Therefore, two ropes of an active weight- 
lifting system support the arm against gravity. The 
robot can be extended by a robotic wrist joint, 
which provides one additional active and two ad- 
ditional passive DOF Force and position sensors 
are integrated to enable admittance control strate- 
gies for interactive support for patient movements. 
The system has been designed for the rehabilita- 
tion of stroke patients. 

One of the most advanced and commonly used 
arm therapy robots is the MIT-Manus (Hogan, 
Krebs, Sharon, & Charnnarong, 1995; Krebs, 
Hogan, Aisen, & Volpe, 1998). It is a planar 
SCARA module that provides two-dimensional 
movements of the patient's hand (figure 22.3). 
Forces and movements are transferred via a robot- 
mounted handle gripped by the patient. The MIT- 
Manus was designed to have a low intrinsic 
end-point impedance (i.e., it is back-drivable) with 
a low inertia and friction. Force and position sen- 
sors are used to feed the impedance controllers. A 
three-DOF module can be mounted on the end of 
the planar module, providing additional wrist mo- 
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Figure 22.3. Patient using the M1T- 
Manus (Hogan et al., 1995; Krebs et al., 

1 



tions in three active DOE Visual movement in- 
structions are given by a graphical display. Clinical 
results with more then 100 stroke patients have 
been published so far (Volpe, Ferraro, Krebs, & 
Hogan, 2002). 

Lum, Burgar, Shor, Majmundar, and van der 
Loos (2002) developed the MIME (Mirror Image 
Movement Enhancer) arm therapy robot. The key 
element of the MIME is a six-DOF industrial robot 
manipulator (Puma 560, Staubli, Inc.) that applies 
forces to a patient's hand that is holding a handle 
connected to the end-effector of the robot. With 
this setup, the forearm can be positioned within a 
large range of spatial positions and orientations. 
The affected arm performs a mirror movement of 
the movement defined by the intact arm. A six-axis 
force sensor and position sensors inside the robot 
allow the realization of four different control 
modes, including position and impedance control 
strategies. Clinical results based on 27 subjects 
have been published so far. 

ARMin is another rehabilitation robot system 
currently being developed at the Swiss Federal 
University of Technology (ETH) and Balgrist Uni- 
versity Hospital, both in Zurich (figure 22.4). The 
robot is fixed at the wall with the patient sitting be- 
neath. The distal part is characterized by an ex- 
oskeleton structure, with the patient's arm placed 
inside an orthotic shell. The current version com- 
prises four active DOF in order to allow elbow flex- 
ion and extension and spatial shoulder movements. 
A vertically oriented, linear motion module per- 
forms shoulder abduction and adduction. Shoulder 



rotation in the horizontal plane is realized by a 
conventional rotary drive attached to the slide of 
the linear motion module. Internal and external 
shoulder rotation is achieved by a special custom- 
made drive that is connected to the upper arm via 
an orthotic shell. Elbow flexion and extension are 
realized by a conventional rotary drive. Several 
force and position sensors enable the robot to work 
with different impedance control strategies. The ro- 
bot can be extended by one additional DOF to al- 
low also hand pronation and supination — an 
important DOF to perform ADLs. The robot is de- 
signed primarily for the rehabilitation of incom- 
plete tetraplegic and stroke patients. It works in 
two different main modes. In the zero-impedance 
mode, the therapist can move the patient's arm to- 
gether with the robot with almost zero resistance. 
This movement is recorded and saved so that it can 
be repeated with any kind of controller, such as a 
position or cooperative controller. 

A review on developments and clinical use of 
current arm therapy robots has been presented by 
Riener, Nef, and Colombo (2005). 

Cooperative Control Strategies 

Many robotic movement trainers do not adapt their 
movement to the activity of the patient. Even if the 
patient is passive, that is, unable to intervene, she 
or he will be moved by the device along a prede- 
fined fixed trajectory. 

Future projects and studies will focus on so- 
called patient-cooperative or subject-centered 
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Figure 22.4. The Zurich arm rehabilitation robot ARMin. 



strategies that will recognize the patient's move- 
ment intention and motor abilities in terms of mus- 
cular efforts, feed the information back to the 
patient, and adapt the robotic assistance to the pa- 
tient's contribution. The best control and display 
strategy will do the same as a qualified human 
therapist — it will assist the patient's movement 
only as much as necessary. This will allow the pa- 
tient to actively learn the spatiotemporal patterns 
of muscle activation associated with normal gait 
and arm/hand function. 

The term cooperativity comprises the meanings 
of compliant, because the robot behaves softly and 
gently and reacts to the patient's muscular effort; 
adaptive, because the robot adapts to the patient's 
remaining motor abilities and dynamic properties; 
and supportive, because the robot helps the patient 
and does not impose a predefined movement or 
behavior. Examples of cooperative control strate- 
gies are, first, impedance control methods that 
make the Lokomat soft and compliant (Hogan 
1985; Riener, Burgkart, Frey, & Proll, 2004); sec- 
ond, adaptive control methods that adjust refer- 
ence trajectory or controller to the individual 
subject (Jezernik, Scharer, Colombo, & Morari, 
2003; Jezernik et al., 2004); and, third, patient- 
driven motion reinforcement methods that support 



patient-induced motions as little as necessary 
(Riener & Fuhr, 1998). 

It is expected that patient-cooperative strate- 
gies will stimulate active participation by the pa- 
tient. They have also the potential to increase the 
motivation of the patient, because changes in mus- 
cle activation will be reflected in the walking pat- 
tern, consistently causing a feeling of success. It is 
assumed that patient-cooperative strategies will 
maximize the therapeutic outcome. Intensive clini- 
cal studies with large patient populations have to 
be carried out to prove these hypotheses. 



Neuroprosthetics 

Background 

Neuroprostheses on the basis of functional electri- 
cal stimulation (FES) may be used to restore motor 
function in patients with upper motor neuron le- 
sions. The underlying neurophysiological principle 
is the generation of action potentials in the unin- 
jured lower motor neurons by external electrical 
stimulation (for review, see Quintern, 1998). 

The possibility of evoking involuntary contrac- 
tions of paralyzed muscles by externally applied 



Neurorehabilitation Robotics and Neuroprosthetics 353 




Figure 22.5. Paraplegic patient with a laboratory neuroprosthesis system applied to stair climbing (T. Fuhr, TU 

Munchen). See also color insert. 



electricity was already known in the 18th century 
(Franklin, 1757). However, after these initial feasi- 
bility demonstrations, more than 200 years were to 
pass until functionally useful movements of para- 
lyzed muscles could be evoked by electrical stimu- 
lation. The first demonstration of standing by FES 
in a spinal cord injury patient was reported by 
Kantrowitz (1960). He applied electrical stimula- 
tion to the quadriceps and gluteus muscles via sur- 
face electrodes. The first portable neuroprosthesis 
for the lower extremities in patients with upper 
motor neuron lesions was developed by Liberson, 
Holmquest, Scot, and Dow (1961). They stimu- 
lated the peroneal nerve with surface electrodes in 
hemiplegic patients to prevent foot drop during the 
swing phase of gait. One decade later, several im- 
plantable FES systems for lower extremity applica- 
tions in hemiplegic patients (Waters, McNeal, & 
Perry, 1975) and paraplegic patients (Brindley, 
Polkey, & Rushton, 1978; Cooper, Bunch, & 
Campa, 1973) were developed and tested. Later, 
several groups derived multichannel neuroprosthe- 
ses with more sophisticated stimulation sequences 
and stimulation via surface electrodes (Kralj, Bajd, & 
Turk, 1980; Kralj, Bajd, Turk, Krajnik, & Benko, 
1983; Malezic et al., 1984) or percutaneous wire 
electrodes (Marsolais & Kobetic, 1987). In the last 
20 years, rapid progress in microprocessor technol- 
ogy has provided the means for computer- 
controlled FES systems (Petrofsky & Phillips, 
1983; Riener & Fuhr, 1998; Thrope, Peckham, & 
Crago, 1985), which enable flexible programming 



of stimulation sequences or even the realization of 
complex feedback (closed-loop) control strategies 
(figure 22.5). 

However, current commercially available neu- 
roprostheses for the lower extremities still work in 
the same fashion as the first peroneal nerve stimu- 
lator (Liberson et al., 1961) or the early multichan- 
nel systems (Kralj et al., 1980, 1983). Whereas 
other neuroprosthetic devices such as the cochlea 
implant, the phrenic pacemaker, and the sacral an- 
terior root stimulator for bladder control have 
grown into reliable, functionally useful, commer- 
cially available neuroprostheses (Peckham et al., 
1996), lower extremity applications are far from 
this stage of development. 

Technical Principles 

Although FES is often referred to as muscle stimula- 
tion, mainly nerve fibers innervating a muscle are 
stimulated, irrespective of the type and localization 
of the electrodes. This, of course, requires that the 
respective lower motor neurons are preserved. Elec- 
trical stimulation activates the motor neurons and 
not the muscle fibers, because the threshold for 
electrical stimulation of the motor axons is far below 
the threshold of the muscle fibers (Mortimer, 1981). 
In neuroprostheses, pulsed currents are ap- 
plied, each pulse releasing a separate action poten- 
tial in neurons, which are depolarized above 
threshold. Not only the current amplitude of the 
externally applied stimulation pulse but also the 
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duration of the pulse, its pulse width, determines if 
a specific neuron is recruited. The threshold value 
above which a neuron is recruited depends on its 
size, the electrical properties of the neuron and 
electrodes, the position of the electrodes relative to 
the neuron, and the type of electrodes. When elec- 
trical pulses of low intensity (low charge per pulse) 
are applied, only large low-threshold neurons 
which are close to the electrodes are recruited. 
With increasing intensity of the pulses, also small 
neurons with higher thresholds and neurons that 
are located further away from the electrodes are re- 
cruited (Gorman & Mortimer, 1983). 

When FES is applied to the neuromuscular 
system, muscle force increases with the number of 
recruited motor units (spatial summation), and 
therefore modulation of pulse width or pulse am- 
plitude can be used to control muscle force (Crago, 
Peckham, & Thrope, 1980; Gorman & Mortimer, 
1983). Another possible method of controlling 
muscle force in FES applications is modulation of 
the stimulation frequency (temporal summation). 
However, the frequency range is limited, as low- 
stimulation frequencies produce unfused single 
twitches rather than a smooth muscular contrac- 
tion (i.e., tetanus). On the other hand, muscle 
force saturates when stimulated with frequencies 
above 30 Hz. With increasing frequencies, the 
muscle is also subjected to fatigue earlier (Brindley 
etal., 1978). 

Stimulation systems and electrodes can be 
grouped into external, percutaneous, and implanted 
systems. In external systems, control unit and stim- 
ulator are outside the body. Surface electrodes are 
used that are attached to the skin above the muscle 
or peripheral nerve, whereas in percutaneous sys- 
tems wire electrodes pierce the skin near the motor 
point of the muscle. In implanted systems, both 
stimulator and electrodes are inside the body. Differ- 
ent kinds of implanted electrodes are used. They 
can be inserted into muscle (e.g., on muscle surface: 
epimysial electrodes), nerve (epineural electrodes), 
or fascicle (intrafascicular electrodes), or surround 
the nerve (nerve cuff electrodes). 

Challenges in the Development 
of Neuroprostheses 

The development of control systems for neuropros- 
theses presents challenges at several different lev- 



els. First, the physiological system that we are try- 
ing to control has many complex features, many of 
which are poorly understood or poorly character- 
ized. Musculoskeletal geometry, dynamic response 
properties of muscle, segmental coupling, reflex in- 
teractions, joint stiffness properties, and nonlinear 
system properties have all caused problems for 
many of the FES control systems that have been 
tested (Adamczyk & Crago, 1996; Chizeck, 1992; 
Hatwell, Oderkerk, Sacher, & Inbar, 1991; Quar- 
tern, 1998; Veltink, Chizeck, Crago, & El-Bialy, 
1992). Perhaps the most important features of the 
physiological system are the high degree of uncer- 
tainty and variability in response properties from 
person to person and the fact that these properties 
change over time due to fatigue and other factors. 
The uncertainty, variability, and time dependence 
make it extremely difficult to determine a stimula- 
tion pattern that will achieve the desired posture or 
movement. 

The second level of challenges includes those 
that are specific to the implementation of FES con- 
trol systems. By far the most prominent of these is- 
sues has been sensors (Crago, Chizeck, Neuman, & 
Hambrecht, 1986). Achieving improved control is 
critically dependent on reliable measurements of 
neuromotor variables in real time. While this has 
occasionally been achieved in laboratory environ- 
ments, it has yet to be achieved in a manner that 
would be suitable for use on an everyday basis. An 
exciting approach that has great potential for solv- 
ing some sensing problems is to record and inter- 
pret signals from intact sensory neurons (Hoffer 
et al., 1996; Yoshida & Horch, 1996). Other 
implementation-level challenges include input de- 
vices, stimulator design, cosmesis, and battery 
weight. 

The third level of challenges includes those re- 
lated to the interactions between three competing 
control systems. The neuroprosthesis control sys- 
tem acts via the electrically stimulated muscles; the 
intact voluntary control system acts via muscles 
that are not paralyzed; and the spinal reflex control 
system (mediated by circuits below the level of the 
lesion) acts via paralyzed muscles. The challenge 
for the neuroprosthesis control system is to act in a 
manner that is coordinated with the voluntary con- 
trol system while effectively exploiting appropriate 
reflexes and counteracting the effects of inappro- 
priate reflexes and spasticity. 
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Approaches to Neuroprosthesis Control 

Current neuroprostheses for the lower extremities 
have not found wide acceptance for clinical use. 
The gain of mobility in terms of walking speed and 
distance is limited. Complex movements with high 
coordination requirements, such as ascending and 
descending stairs, are impossible as yet. The reason 
for this limited function is that all commercially 
available systems are open-loop systems, which do 
not provide sensor feedback to determine the stim- 
ulation pattern. 

The block diagram in figure 22.6 demonstrates 
the various types of control that have been used in 
FES systems (see also Abbas & Riener, 2001). To 
operate FES systems, users provide inputs that are 
either discrete selections or continuously variable 
signals. Discrete input signals can be used to select 
a task option or to trigger the initiation of a move- 
ment pattern. A continuously variable signal can be 
used to adjust stimulation patterns in a continuous 
manner as the task is being performed. This signal, 
sometimes called a command input, can be used to 
adjust the stimulation to several muscles simulta- 
neously using a nonlinear mapping function 
(Adamczyk & Crago, 1996; Peckham, Keith, & 
Freehafer, 1988). Most FES systems use the input 
signals in an open-loop control system configura- 
tion, which means that the action of the controller 



will be the same each time the user gives the input 
(Hausdorff & Durfee, 1991; McNeal, Nakai, Mead- 
ows, & Tu, 1989). For example, in lower-extremity 
FES systems for standing up, each time the user 
gives the discrete input signal to stand up, a pre- 
specified pattern of stimulation is delivered to a set 
of muscles (Kobetic & Marsolais, 1994). In upper- 
extremity systems for hand grasp, each time the 
user changes the level of command input, the stim- 
ulation levels delivered to a set of muscles are ad- 
justed using a prespecified mapping function 
(Peckham et al., 1988). In either case, the relation- 
ship between the input signal and the stimulation 
must be predetermined in a process that can be de- 
scribed as fitting the FES control system to the 
user. This process is typically time consuming and 
requires the effort of a trained rehabilitation team. 

The effectiveness of an open-loop approach is 
often limited by the ability of the rehabilitation 
team to predict or anticipate the response of the 
musculoskeletal system to the stimulation. If these 
predetermined stimulation patterns are not appro- 
priate, then the desired posture or movement may 
not be achieved. Furthermore, as muscles fatigue 
and system properties change over time, a repeti- 
tion of the fitting process may be required. In sys- 
tems that use continuously variable inputs, the 
user can often observe system performance and 
make online adjustments as needed. This approach 
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Figure 22.6. Block diagram representation of the neuroprosthesis control components. The control 
system may include feedforward, feedback, or adaptive components. Note that if feedforward control 
is used alone, it is usually referred to as open-loop control. 
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takes advantage of the intact sensory functions of 
the user (often visual observations), but it may put 
excessive demands on the attention of the user. 

In order to improve the performance of these 
systems, advanced controllers must be designed 
that are capable of determining appropriate stimu- 
lation levels to accomplish a given task. The most 
commonly used approach to improving the quality 
of the control system has been to utilize feedback 
or closed-loop control strategies (see figure 22.6). 
Closed-loop control means that information re- 
garding the actual state of the system (e.g., body 
posture and ground reaction forces) is recorded by 
sensors and fed back to a controller. Based on the 
measured signals, the controller then determines 
the stimulation pattern that is required to achieve a 
specific movement task. This type of control mim- 
ics the online adjustments that can be made by the 
user but does so in a manner that is automatic and 
does not require conscious effort on the part of the 
user. Feedback control can be used to supplement 
the signals from the open-loop controller and may 
be able to improve performance by adjusting the 
stimulation to account for inaccuracies in the 
open-loop fitting process. In addition, external 
and internal disturbances can be recognized and 
the stimulation pattern readjusted to result in a 
smooth and successful movement. Most closed- 
loop systems have been evaluated for control of 
force or angle at single joints (Hatwell et al., 1991; 
Munih, Donaldson, Hunt, & Barr, 1997; Veltink 
et al., 1992; Yoshida & Horch, 1996). Little work 
has been done in the field of closed-loop control of 
multijoint movements such as standing (Jaeger, 
1986), standing up and sitting down (Mulder, 
Veltink, & Boom, 1992), and walking (Fuhr, Quin- 
tern, Riener, & Schmidt, 2001). 

One promising approach to improving lower- 
extremity systems is the use of continuously variable 
inputs from the user in a manner that is similar to the 
command input used in upper-extremity systems 
(Donaldson & Yu, 1996; Riener & Fuhr, 1998; 
Riener, Ferrarin, Pavan, & Frigo, 2000). Such strate- 
gies are called patient-driven or subject-centered 
strategies, because the person drives the movement, 
in contrast to controller-centered approaches, where 
a predefined reference signal is used. 

Whenever a continuous input signal is em- 
ployed, it can be difficult for the user to make 
adjustments when the system input-output proper- 
ties are unpredictable. Adaptive control strategies 



have been developed to adjust the overall system 
behavior (i.e., the response of the combined con- 
troller and system) so that it is more linear, repeat- 
able, and therefore predictable (Adamczyk & 
Crago, 1996; Kataria & Abbas, 1999). These tech- 
niques adjust the parameters of the control system 
and attempt to self-fit the system to the user in or- 
der to make it easier to use and easier to learn to 
use (Chang et al., 1997; Chizeck, 1992; Crago, 
Lan, Veltink, Abbas, & Kantor, 1996). 

The performance of closed-loop approaches is 
still not satisfactory in terms of disturbance com- 
pensation, upper body incorporation, variable step 
adjustment, or movement smoothness. This is one 
of the main reasons why current systems are not 
applied clinically so far. The use of computational 
models can significantly enhance the design and 
test of closed-loop control strategies applied to FES 
(see Riener, 1999). Time-consuming and perhaps 
troublesome trial-and-error experimentation can 
be avoided or at least shortened, and the number 
of experiments with humans can be reduced, both 
of which can accelerate the development of neuro- 
prostheses. Furthermore, physiologically based 
mathematical models can provide significant in- 
sight into relevant activation and contraction pro- 
cesses. This insight may help us to better 
understand and eventually avoid the disadvanta- 
geous effects occurring during FES, such as in- 
creased muscular fatigue. Eventually, muscle force 
production and the resulting movement may be 
optimized to obtain better functionality. 



Summary 

After a general overview of the principle of human 
motion generation and related pathologies, two ex- 
amples of movement restoration were presented in 
more detail. In the first example, it was shown how 
robots can be applied to support natural restora- 
tion principles of the CNS. In the second example, 
the technical principles and challenges of neuro- 
prostheses were presented. 



Main Points 

1 . The damaged nervous system can recover by 
application of natural and artificial restoration 
principles. 
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2. Robots can make motion therapy more 
efficient by increasing training duration and 
number of training sessions, while reducing 
the number of therapists required per patient. 

3. Patient-cooperative control strategies have the 
potential to further increase the efficiency of 
robot-aided motion therapy 

4. Neuroprostheses can restore movement in 
patients with upper motor neuron lesions. 

5. Neuroprosthesis function can be improved by 
applying closed-loop control components and 
computational models. 



Key Readings 

Abbas, J., & Riener, R. (2001). Using mathematical 
models and advanced control systems techniques 
to enhance neuroprosthesis function. Neuromodu- 
lator!, 4, 187-195. 

Quintern,J. (1998). Application of functional electrical 
stimulation in paraplegic patients. Neurorehabilita- 
tion, 10, 205-250. 

Riener, R. (1999). Model-based development of neu- 
roprostheses for paraplegic patients. Royal Philo- 
sophical Transactions: Biological Sciences, 354, 
877-894. 

Riener, R., Nef, X, & Colombo, G. (2005). Robot- 
aided neurorehabilitation for the upper extremi- 
ties. Medical and Biological Engineering and 
Computing, 43, 2-10. 

Riener, R., Lunenburger, L., Jezernik, S., Anderschitz, 
M., Colombo, G. & Dietz, V (2005). Cooperative 
subject-centered strategies for robot-aided tread- 
mill training: first experimental results. IEEE Trans- 
actions on Neural Systems and Rehabilitation 
Engineering 13, 380-393. 



References 

Abbas, J., & Riener, R. (2001). Using mathematical 
models and advanced control systems techniques 
to enhance neuroprosthesis function. Neuromodu- 
lation, 4, 187-195. 

Adamczyk, M. M., & Crago, P. E. (1996). Input- 
output nonlinearities and time delays increase 
tracking errors in hand grasp neuroprostheses. 
IEEE Transactions on Rehabilation Engineering, 4, 
271-279. 

Barbeau, H., & Rossignol, S. (1994). Enhancement of 
locomotor recovery following spinal cord injury. 
Current Opinion in Neurology, 7, 517-524. 



Brindley, G. S., Polkey, C. E., & Rushton, D. N. (1978). 
Electrical splinting of the knee in paraplegia. Para- 
plegia, 6, 428-435. 

Chang, G. C, Luh.J.J., Liao, G. D., Lai,J. S., Cheng, 
C. K., Kuo, B. L, et al. (1997). A neuro-control 
system for the knee joint position control with 
quadriceps stimulation. IEEE Transactions on Reha- 
bilation Engineering, 5, 2-11. 

Chizeck, H. J. (1992). Adaptive and nonlinear control 
methods for neural prostheses. In R. B. Stein, P H. 
Peckham, & D. B. Popovic (Eds.), Neural prosthe- 
ses: Replacing motor function after disease or disability 
(pp. 298-328). New York: Oxford University 
Press. 

Colombo, G., Jorg, M., & Jezernik, S. (2002). Automa- 
tisiertes Lokomotionstraining auf dem Laufband. 
Automatisierungstechnik, 50, 287-295. 

Colombo, G, Jbrg, M., Schreier, R., & Dietz, V (2000). 
Treadmill training of paraplegic patients using a 
robotic orthosist. Journal of Rehabilitation Research 
and Development, 37, 693-700. 

Cooper, E. B., Bunch, W H., & Campa, J. H. (1973). 
Effects of chronic human neuromuscular stimula- 
tion. Surgery Forum, 24, 477-479. 

Cozens, J. A. (1999). Robotic assistance of an active 
upper limb exercise in neurologically impaired pa- 
tients. IEEE Transactions on Rehabilation Engineer- 
ing, 7, 254-256. 

Crago, P E., Chizeck, H. J., Neuman, M. R., & Ham- 
brecht, E T. (1986). Sensors for use with func- 
tional neuromuscular stimulation. E Transactions on 
Biomedical Engineering, 33, 256-268. 

Crago, P E., Lan, N., Veltmk, P. H., Abbas, J. J., & Kan- 
tor, C. (1996). New control strategies for neuro- 
prosthetic systems. Journal of Rehabilation Research 
and Development, 33, 158-172. 

Crago, P E., Peckham, P H., & Thrope, G B. (1980). 
Modulation of muscle force by recruitment during 
intramuscular stimulation. IEEE Transactions on 
Biomedical Engineering, 27, 679-684. 

Dietz, V, Colombo, G, & Jensen, L. (1994). Locomo- 
tor activity in spinal man. Lancet, 44, 1260-1263. 

Donaldson, N. de N., & Yu, C.-H. (1996). FES stand- 
ing control by handle reactions of leg muscle stim- 
ulation (CHRELMS) . IEEE Transactions on 
Rehabilation Engineering, 4, 280-284. 

Franklin, B. (1757). On the effects of electricity in para- 
lytic cases. Philosophical Transactions, 50, 481-483. 

Fuhr, T., Quintern, J., Riener, R., & Schmidt, G. 
(2001). Walk! Experiments with a cooperative 
neuroprosthetic system for the restoration of gait. 
In Ron J. Triolo (Ed.),Proceedings of the IFESS Con- 
ference (pp. 46-47). Cleveland OH, June.. Pub- 
lished by Dept. of Orthopaedics and Biomedical 
Engineering, Case Western Reserve University, and 
Louis Stokes Veterans Affairs Medical Center. 



358 



Special Populations 



Gorman, P H., & Mortimer, J. T. (1983). The effect of 
stimulus parameters on the recruitment character- 
istics of direct nerve stimulation. IEEE Transactions 
on Biomedical Engineering, 30, 301-308. 

Harwin, W, Loureiro, R., Amirabdollahian, E, Taylor, 
M.Johnson, G., Stokes, E., et al. (2001). The 
GENTLE/s project: A new method for delivering 
neuro-rehabilitation. In C. Marincek et al. (Eds.), 
Assistive technology — added value to the quality of 
life. AAATEV1 (pp. 36-41). Amsterdam: IOS 
Press. 

Hatwell, M. S., Oderkerk, B. J., Sacher, C. A., & Inbar, 
G. E (1991). The development of a model refer- 
ence adaptive controller to control the knee joint 
of paraplegics. IEEE Transactions on Automatic Con- 
trol, 36, 683-691. 

Hausdorff, J. M., & Durfee, W K. (1991) Open-loop 
position control of the knee joint using electrical 
stimulation of the quadriceps and hamstrings. 
Medical and Biological Engineering and Computing, 
29, 269-280. 

Hesse, S., Bertelt, C, Jahnke, M. T, Schaffrin, A., 
Baake, P., Malezic, M., et al. (1995). Treadmill 
training with partial body weight support com- 
pared with physiotherapy in nonambulatory hemi- 
paretic patients. Stroke, 26, 976-981. 

Hesse, S., Schulte-Tigges, G, Konrad, M., Bardeleben, 
A., & Werner, C. (2003). Robot-assisted arm 
trainer for the passive and active practice of bilat- 
eral forearm and wrist movements in hemiparetic 
subjects. Archives of Physical Medicine and Rehabili- 
tation, 84, 915-920. 

Hesse, S., & Uhlenbrock, D. (2000). A mechanized 
gait trainer for restoration of gait. Journal of Reha- 
bilitation Research and Development, 37, 701-708. 

Hoffer, J. A., Stein, R. B., Haugland, M. K., Sinkjaer, T, 
Durfee, W K., Schwartz, A. B., et al. (1996). 
Neural signals or command control and feedback 
in functional neuromuscular stimulation: A review. 
Journal of Rehabilitation Research and Development, 
33, 145-157. 

Hogan, N. (1985). Impedance control: An approach to 
manipulation, Parts I, II, III. Journal of Dynamic 
Systems, Measurement, and Control, 107, 1-23. 

Hogan, N., Krebs, H. I., Sharon, A., & Charnnarong, J . 
(1995). Interactive robotic therapist. U.S. Patent 
5466213. 

Jaeger, R. J. (1986). Design and simulation of closed- 
loop electrical stimulation orthoses for restoration 
of quiet standing in paraplegia. Journal of Biome- 
chanics, 19, 825-835. 

Jezemik, S., Colombo, G., & Morari, M. (2004). Auto- 
matic gait-pattern adaptation algorithms for reha- 
bilitation with a 4 DOF robotic orthosi. IEEE 
Transactions on Robotics and Automation, 20, 
574-582. 



Jezemik, S., Scharer, R., Colombo, G, & Morari, M. 
(2003). Adaptive robotic rehabilitation of locomo- 
tion: A clinical study in spinally injured individu- 
als. Spinal Cord, 41(12). 

Kantrowitz, A. (1960). Electronic physiologic aids. In 
Report of the Maimonides Hospital (pp. 4-5). Un- 
published report, Maimonides Hospital, Brooklyn, 
NY. 

Kataria, P, & Abbas, J. J. (1999). Adaptive user- 
specified control of movements with functional 
neuromuscular stimulation. Proceedings of the 
1EEE/BMES Conference (Atlanta, GA, p. 604). 

Kobetic, R., & Marsolais, E. B. (1994). Synthesis of 
paraplegic gait with multichannel functional neu- 
romuscular stimulation. IEEE Transactions on Reha- 
biltation Engineering, 2, 66-79. 

Krai], A., Bajd, T, & Turk, R. (1980). Electrical stimu- 
lation providing functional use of paraplegic pa- 
tient muscles. Medical Progress Through Technology, 
7, 3-9. 

Kralj, A., Bajd, T, Turk, R., Krajnik, J., & Benko, H. 
(1983). Gait restoration in paraplegic patients: A 
feasibility demonstration using multichannel sur- 
face electrode FES. Journal of Rehabilitation Re- 
search and Development, 20, 3-20. 

Krebs, H. 1., Hogan, N., Aisen, M. L., & Volpe, B. T 
(1998). Robot-aided neurorehabilitation. IEEE 
Transactions on Rehabilation Engineering, 6, 75-87. 

Kwakkel, G., Wagenaar, R. C, Koelman, T. W., 

Lankhorst, G J., & Koetsier, J. C. (1997). Effects 
of intensity of rehabilitation after stroke: A re- 
search synthesis. Stroke, 28, 1550-1556. 

Liberson, W. T., Holmquest, M. E., Scot, D., & Dow, 
M. (1961). Functional electrotherapy: Stimulation 
of the peroneal nerve synchronized with the swing 
phase of gait of hemiplegic patients. Archives of 
Physical Medicine and Rehabilitation, 42, 101-105. 

Lum, P S., Burgar, C. G, Shor, P C, Majmundar, M., 
& van der Loos, M. (2002). Robot-assisted move- 
ment training compared with conventional therapy 
techniques for the rehabilitation of upper-limb 
motor function after stroke. Archives of Physical 
Medicine and Rehabilitation, 83, 952-959. 

Malezic, M., Stanic, U., Kfjajic, M., Acimovic, R., Kra- 
jnik, J., Gros, N., et al. (1984). Multichannel elec- 
trical stimulation of gait in motor disabled 
patients. Orthopaedics, 7, 1187-1195. 

Marsolais, E. B., & Kobetic, R. (1987). Functional elec- 
trical stimulation for walking in paraplegia. Journal 
of Bone and Joint Surgery, 69-A, 728-733. 

McNeal, D. R., Nakai, R. J., Meadows, R, & Tu, W 
(1989). Open-loop control of the freely swinging 
paralyzed leg. IEEE Transactions on Biomedical Engi- 
neering, 36, 895-905. 

Mortimer, J. T. (1981). Motor prostheses. InV B. 

Brooks (Ed.), Handbook of physiology , nervous system 



Neurorehabilitation Robotics and Neuroprosthetics 



359 



II (pp. 155-187). Bethesda, MD: American Physio- 
logical Society. 

Mulder, A. J., Veltink, P H., & Boom, H. B. K. (1992). 
On/off control in FES-induced standing up: A 
model study and experiments. Medical and Biologi- 
cal Engineering and Computing, 30, 205-212. 

Munih, M., Donaldson, N. de N., Hunt, K. J., & Barr, 
E M. D. (1997). Feedback control of unsupported 
standing in paraplegia — part IP Experimental re- 
sults. IEEE Transactions on Rehabilation Engineering, 
5, 341-352. 

Peckham, P H., Keith, M. W, & Freehafer, A. A. 

(1988). Restoration of functional control by elec- 
trical stimulation in the upper extremity of the 
quadraplegic patient. Journal of Bone Joint Surgery, 
70-A, 144-148. 

Peckham, P H., Thrope, G., Woloszko, J., Habasevich, 
R., Scherer, M., & Kantor, C. (1996). Technology 
transfer of neuroprosthetic devices. Journal of Re- 
habilitation Research and Development, 33, 
173-183. 

Petrofsky, J. S., & Phillips, C. A. (1983). Computer 
controlled walking in the paralyzed individual. 
Journal of Neurological and Orthopedic. Surgery, 4, 
153-164. 

Platz, T. (2003). Evidenzbasierte Armrehabilitation: 
Eine systematische Literaturubersicht. Nervenarzt, 
74, 841-849. 

Quintern, J. (1998). Application of functional electrical 
stimulation in paraplegic patients. Neurorehabilita- 
tion, 10, 205-250. 

Reinkensmeyer, D. J., Wynne, J. H., & Harkema, S.J. 
(2002). A robotic tool for studying locomotor 
adaptation and rehabilitation. Second Joint Meeting 
of the IEEE, EMBS and BMES 2002 (pp. 
2353-2354). 

Riener, R. (1999). Model-based development of 
neuroprostheses for paraplegic patients. Royal 
Philosophical Transactions: Biological Sciences, 354, 
877-894. 

Riener, R., Burgkart, R., Frey M., & Proll, T. (2004). 
Phantom-based multimodal interactions for med- 
ical education and training: The Munich Knee 



Joint Simulator. IEEE Transactions on Information 
Technology in Biomedicine, 8, 208-216. 

Riener, R., Ferrarin, M., Pavan, E., & Frigo, C. (2000). 
Patient-driven control of FES-supported standing 
up and sitting down: Experimental results. IEEE 
Transactions on Rehabilitation Engineering, 8, 
523-529+. 

Riener, R., & Fuhr, T. (1998). Patient-driven control of 
FES-supported standing-up: A simulation study 
IEEE Transactions on Rehabilitation Engineering, 6, 
113-124. 

Riener, R., Nef, T, & Colombo, G. (2005). Robot- 
aided neurorehabilitation for the upper extremi- 
ties. Medical and Biological Engineering and 
Computing, 43, 2-10. 

Shumway-Cook, A., & Woollacott, M. H. (2001). Mo- 
tor control: Theory and practical applications (2nd 
ed.). Baltimore: Lippincott Williams and Wilkins. 

Thrope, G. B., Peckham, P. R, & Crago, P E. (1985). A 
computer controlled multichannel stimulation sys- 
tem for laboratory use in functional neuromuscu- 
lar stimulation. IEEE Transactions on Biomedical 
Engineering, 32, 363-370. 

Van der Linde, R. Q., Lammertse, P., Frederiksen, E., & 
Ruiter, B. (2002). The HapticMaster, a new high- 
performance haptic interface. In Proceedings ojEu- 
rohapti.cs, Edinburgh, UK (pp. 1-5). 

Veltink, P. H., Chizeck, H. J., Crago, P E., & El-Bialy, 
A. (1992). Nonlinear joint angle control for artifi- 
cially stimulated muscle. IEEE Transactions on Bio- 
medical Engineering, 39, 368-380. 

Volpe, B. T., Ferraro, M., Krebs, H. I., & Hogan, N. 
(2002). Robotics in the rehabilitation treatment of 
patients with stroke. Current Atherosclerosis Reports, 
4, 270-276. 

Waters, R. L, McNeal, D. R., & Perry, J. (1975). Exper- 
imental correction of footdrop by electrical stimu- 
lation of the peroneal nerve. Journal of Bone and 
Joint Surgery, 57-A, 1047-1054. 

Yoshida, K, & Horch, K. (1996). Closed-loop control 
of ankle position using muscle afferent feedback 
with functional neuromuscular stimulation. IEEE 
Transactions on Biomedical Engineering, 43, 167-176. 




Matthew Rizzo, Sean McEvoy, and John Lee 



Medical Safety and 
Neuroergonomics 



Primum non noscere: First, do no harm. 
Physicians' credo 

Errare humanum est: To err is human. 

Seneca 



Errors in medicine are an important public health 
policy issue that can be mitigated by applying prin- 
ciples and techniques of neuroergonomics. The In- 
stitute of Medicine (IOM, 2000) issued a 
publication, To Err Is Human: Building a Safer Health 
System, which asserted that errors in health care are 
a leading cause of death and injury, killing more 
people than do car crashes, AIDS, or breast cancer. 
This IOM report reviewed the frequency, cost, and 
public perceptions of safety errors and suggested 
that between 44,000 and 98,000 deaths per year 
result from medical errors. These figures were ex- 
trapolated from a 1984 study of New York and a 
1992 study of Colorado and Utah. Whether these 
limited samples truly reflect what goes on in the 
huge and variegated U.S. medical population is un- 
clear. Nevertheless, medical errors are clearly a 
public health problem and systems should be de- 
veloped to mitigate error- related injuries and 
death — as they were decades ago in other indus- 
tries for which safety is of critical importance, such 
as aviation and nuclear power. 

This chapter considers how neuroergonomics, 
the study of the brain and behavior at work in 
healthy and impaired states, is relevant to assess- 
ments and interventions in patient safety at the 
levels of individuals and health care systems. For 



example, knowledge of how the brain processes 
visual, auditory, and tactile information can pro- 
vide guidelines and constraints for theories of in- 
formation presentation and health care task and 
system designs. It is particularly important to con- 
sider alternative approaches because attempts to 
improve safety in response to the IOM report have 
not been particularly successful. For instance, 
computerized physician order entry (CPOE) sys- 
tems, aimed at reducing errors caused by illegibly 
written or ill-advised prescriptions, have limited 
utility in preventing adverse drug effects (Nebeker, 
Hoffman, Weir, Bennett, & Hurdle, 2005) and 
may even exacerbate medication errors (Koppel et 
al, 2005). One widely used CPOE system facili- 
tated 22 types of medication error risks; among 
these were fragmented CPOE displays that pre- 
vented a coherent view of patient medications, 
pharmacy inventory displays mistaken for dosage 
guidelines, ignored antibiotic renewal notices 
placed on paper charts but not in the CPOE sys- 
tem, and inflexible ordering formats generating 
wrong orders. There are clearly many more oppor- 
tunities for reducing errors in the delivery of 
health care and improving patient safety through 
systemwide assessments of health care delivery 
processes. 
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Safe delivery of health care can draw from 
many of the approaches, strategies, and techniques 
outlined in other chapters of this book. For exam- 
ple, patient safety efforts can benefit from insights 
on augmented reality (see chapter 17, this volume) 
by using interactive projection systems that allow 
a surgeon to "see" a patient's anatomy projected on 
a visual model of the patient's skin. Continuous 
monitoring of a physician's physiological state may 
enable systems to modulate the amount of infor- 
mation that goes to a physician, say an anesthesiol- 
ogist, to make sure he or she does not get overloaded 
or confused, or to alert others to the physician's 
state. Other systems could help track patients at 
each step of the health care system to improve situ- 
ation awareness of medical personnel interacting 
with patients and each other in complex settings 
like a busy hospital ward, emergency room, or day- 
of-surgery lounge. This chapter reviews potential 
areas for neuroergonomic interventions at the level 



of individuals and systems, and cultural and legal 
issues that affect the ability to intervene. 



Systems Perspective 

Often the health care system is unsafe, not the 
practitioner or equipment (Bogner, 1994; Woods, 
2000; Woods, Johannesen, Cook, & Sarter, 1994). 
Because many levels of distraction and responsibil- 
ity affect health care providers, a systems-based ap- 
proach is needed to locate the precursors of error 
and identify effective mitigation strategies. Figure 
23.1 (Rasmussen, Pejtersen, & Goodstein, 1994) 
shows how the behavioral sequence that leads to 
errors depends on a complex context that includes 
the task situation and mental strategies, as well as 
the management structure and work preferences of 
individuals. To date, a systems-based approach to 
error assessment has been limited by inadequate 
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Figure 23.1. A systems-based approach to identifying the behavioral sequences that lead to errors. From 
Rasmussen et al. (1994). 
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systems for reporting and classifying errors in 
health care. Reliable and accurate health care error 
databases generally do not exist, due in part to cul- 
ture, fear of reprisal and litigation, and ambiguity 
on what constitutes an error. 



Relationships Between Health Care 
Delivery and Errors 

Relationships between health care delivery and 
safety errors can be represented by an imaginary 
triangle (Heinrich, Petersen, & Roos, 1980) or 
"iceberg" (Maycock, 1997; figure 23.2). The simple 
model can be applied to errors that lead to injuries 
in diverse settings, including factories, automobile 
driving, aviation, nuclear power, and health care. 
Visible safety errors ("above the waterline") include 
errors resulting in fatality, serious injury, mild in- 
jury, or legal claims. While the number of fatalities 
and injuries from health care errors may be unac- 
ceptably high (IOM, 2000), these events are rela- 
tively infrequent. Submerged below the waterline 
are relatively benign mishaps and near misses that 
are theoretically related to injury outcome and oc- 
cur more frequently. 

With sufficient numbers of observations, it 
might be possible to accurately estimate the risk of 
a fatality (a low-frequency, high-severity event) 



Fatalities, 
Serious injury 



Minor 
Injuries 




Near 
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Figure 23.2. Health care errors that lead to fatality or 
serious injury represent only a small portion of health 
care errors. The majority of errors lead only to mild 
consequences or have no direct effect on the patient 
(near misses). 



through the assessment of measurable safety errors 
(high-frequency, low-severity events). The relation- 
ship between these low-frequency, high-severity 
events that result in reported injuries and high- 
severity, low-frequency events that are neither sys- 
tematically observed nor reported might be better 
defined using a system for naturalistic observations 
to record continuous behavior and the patterns of 
performance and circumstances leading to adverse 
events (as described in chapter 8, this volume). 



Cognitive Factors 

Risk of human errors in complex systems such as 
health care increases with distraction, workload, fa- 
tigue, licit and illicit drug use, illness, and associated 
impairments of attention, perception, response se- 
lection (which depends on memory, reasoning, and 
decision making), and implementation (see figure 
23.3). Some medical errors can be detected because 
personnel normally monitor their performance and 
will often detect discrepancies when feedback on 
performance fails to match expectations based on 
correctly formulated intentions. 

In this heuristic framework, the health care 
practitioner (1) perceives and attends stimulus or 
situation evidence and interprets the situation to 
arrive at a diagnosis; (2) formulates a plan based on 
the particular health care situation and relevant 
previous experience or memory; and (3) executes 
an action (e.g., by ordering laboratory tests, med- 
ications, or making referrals to additional practi- 
tioners). The outcome is safe or unsafe due to 
errors at one or more stages, and depends not only 
upon the practitioner but also on the engineering 
of the health care delivery system. The outcome of 
the behavior provides a source of potential feed- 
back for the practitioner to take subsequent action. 

In some cases, the feedback loops of errors in 
health care are short and direct, such as the sound of 
an alarm signifying critical alteration of vital signs 
during a surgical procedure on a patient under 
general anesthesia. In medical subspecialties such 
as internal medicine or neurology, however, these 
feedback loops are often indirect and have a long la- 
tency before they return to the operator or operators. 
In this situation, a system of reporting and classify- 
ing errors is especially important. An error database 
would provide a more global level of understanding 
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Figure 23.3. Information-processing model for understanding practitioner error. 



about the types of errors that occur during extended 
treatment and individual and systematic variables 
that contribute to errors, and help to focus interven- 
tion efforts where they are most needed. 



most medical specialties. Frequency of errors is not 
known and may not be knowable. In the absence of 
these data, useful evidence for directing health care 
safety interventions comes from malpractice claims. 



The Need to Track Health Care Errors 

Mitigating safety errors in medicine depends on 
knowing the type, frequency, and severity of the er- 
rors that occur, as well as what actions lead to 
successful outcomes and in what particular circum- 
stances. Error-reporting systems and a taxonomy or 
lexicon of errors are thus necessary. Analysis of error 
reports can identify the cognitive or organizational 
stresses that contributed to the error and suggest 
mitigation strategies to relieve these stresses. 

Systems for reporting and classifying errors in 
health care are currently inadequate. Consequently, 
proposed interventions are guided by the best avail- 
able evidence, which is often limited. Most errors in 
medical practice are reported at local levels, as with 
incident reports of nursing or medication error at 
hospitals, or in morbidity and mortality rounds, in 
which health care personnel (especially physicians) 
discuss complications of patient care and how to 
improve related procedures and practice. These re- 
ports are not systematically examined, and the 
analysis is not disseminated broadly. Any lessons 
learned from the local analysis of errors are con- 
fined to a few people and do not reach the larger 
organization. Reliable and valid error reporting, 
analysis, and dissemination systems do not exist in 



Closed Claims Analyses 

Malpractice data can serve as a surrogate for identi- 
fying severe medical errors at the tip of the iceberg 
(figure 23.2). Along these lines, Glick (2001) sum- 
marized data from available Massachusetts closed 
malpractice claims involving neurological problems. 
Errors were classified as failure to diagnose, act, or 
decide (e.g., delay in ordering studies, failure to per- 
form a proper history or physical examination, and 
misinterpretation of studies). The basic premise is 
that malpractice claims are surrogates for poor prac- 
tices, bad outcomes, errors, and miscommunication 
and can indicate the need for modifying health care 
systems and educational programs. Glick proposed 
that the information provided valuable lessons for 
neurological teachers on what to teach and whom 
to teach. Overall, there were approximately 150 
cases involving 250 neurological defendants. Find- 
ings showed that the main errors were diagnostic 
failure in one third and treatment failure (especially 
medication errors and professional behavior and 
communication problems including improper con- 
sent) in one third of all cases. Among the diagnostic 
failures were failure to diagnose stroke and other 
vascular problems, spinal cord and nerve root 
problems, meningitis, encephalitis, head injury, and 
brain tumors. About two thirds of these problems 
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were acute. A review of malpractice claims, inpa- 
tient incident reports and chart reviews, and jour- 
nal literature (Glick, 2005) reaffirmed that failure of 
accurate and timely diagnosis was a leading cate- 
gory of neurological health care error. 

These data demonstrate the potential utility of 
error analysis, although the available data are lim- 
ited. It remains difficult to extract detailed charac- 
teristics of individual and systemic performance that 
led to litigation, because insurers' data collection 
systems are not designed for this use. Importantly 
malpractice claims may not accurately reflect med- 
ical error, but may reflect factors unrelated to physi- 
cian competence, such as tone of voice (Ambady, 
LaPlante, Nguyen, Rosenthal, Chaumeton, & Levin- 
son, 2002). A need remains for more serviceable 
sources of error data. 

Mandatory Reporting 

One obstacle to creating a database of errors for 
tracking is bias in reporting. One potential source of 
more reliable data would be a mandatory reporting 
system that requires health care personnel to report 
all medical errors. Yet such systems run against the 
cultural and ethical norms of many Americans and 
may not lead to mandatory reporting by healthcare 
personnel. Along these lines, the State of California 
Health and Safety Code mandates that all physicians 
report immediately the identity of every driver diag- 
nosed with a lapse of consciousness or an episode of 
marked confusion due to neurological disorders or 
senility. The mandatory report triggers an evaluation 
of the individual, during which driving privileges 
may be suspended, but may discourage patients 
with treatable forms of mental impairment to avoid 
evaluation for fear of losing the license to drive 
(Drachman, 1990). A practice among many Califor- 
nia physicians is to inform the patient and patients 
family of concerns about driving with dementia but 
not to report the names of drivers with dementia to 
the state. No reliable or fair means of dealing with 
nonreporters has been devised. Contrary to tradi- 
tional views of epidemiology and teaching on qual- 
ity control, mandatory reporting may not generate 
objective measurements to track. 

Voluntary Reporting 

To provide an accurate source of health care error 
information, it may be necessary to establish a 



health care information safety report system (HISRS) 
to gather voluntary reports from health care work- 
ers, protect the identity of the error reporters, and 
provide safeguards against the use of the error in- 
formation during litigation proceedings. This type 
of protected voluntary reporting system exists in 
other fields where analysis of safety errors carries 
high importance. 

One such program is the Aviation Safety Re- 
porting System (ASRS). The system depends on at- 
tempts to understand the most we can from a 
smattering of reports. Close analysis of the details 
of these reports can remove some or all reporter 
bias. The ASRS is funded by the Federal Aviation 
Administration (FAA) and operated by NASA. Be- 
cause the ASRS is operated by an independent 
agency, it has no licensing or legislative power. 
ASRS has no authority to direct that action be 
taken in potentially hazardous conditions or situa- 
tions, but alerts those (FAA personnel) responsible 
for correcting the conditions. The ASRS acts di- 
rectly and indirectly to inform pilots, flight crews, 
air-traffic controllers, airport managers, and others 
in the aviation community about potentially unsafe 
practices. In certain situations (such as "altitude 
busts"), there is an incentive for pilots to report un- 
safe practices to ASRS to avoid penalty. Individuals 
who report these incidents are granted confiden- 
tiality and immunity. 

Also, reporters are granted use immunity under 
FAR [Federal Aviation Rule] 91.25 which prohibits 
the FAA from using reports submitted under ASRS, 
or information derived from these reports, in any 
enforcement actions against reporters, provided 
that criminal offenses and accidents are not in- 
volved. FAAs Advisory Circular 00-46 also provides 
for limited disciplinary immunity for pilots in con- 
junction with errors resulting from actions or lack 
of actions, if the certain criteria are met. The trans- 
actional immunity of ASRS is a powerful incentive 
for reporting an unintentional violation of FAA 
rules because it "inoculates" the reporting person 
against adverse certificate action or civil penalties. 

Along these lines, in March 2002 the NASA 
ASRS model was adopted by the Veterans Adminis- 
tration (VA) health care system in its Patient Safety 
Reporting System. The VA can implement such a 
system with relative ease because of unique legal 
protections for VA quality assurance activities. 
The VA model uses reports that are voluntary and 
confidential to begin with and, in later stages, are 
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deidentified. The analyst can call the reporter back 
for further information and a better understanding 
of the mechanisms underlying an unsafe occur- 
rence. The courts so far appear to have ruled that the 
deidentified database is hearsay and inadmissible, 
though it may be possible to identify some reports 
in the database for use in adversarial proceedings. 

A significant obstacle to obtaining accurate 
data on health care errors, even within a voluntary 
reporting structure, has been the current lack of 
safe harbors. Voluntary reporting may be prefer- 
able to mandatory reporting but could still pro- 
duce discoverable evidence that fuels litigation 
without illuminating the fundamental causes of 
medical errors. Persons who are aware of data on 
medical errors may not be reporting the data for 
fear of retribution to themselves or colleagues in 
terms of professional sanctions, civil or criminal li- 
ability, or economic loss. Physicians are reluctant to 
share data that has traditionally been managed in 
an adversarial manner. A culture change is required 
so that reporting is voluntary, viewed as supportive 
and part of a process of quality improvement. 

Legislative actions must be taken to ensure the 
legal protections required to introduce such an 
error-tracking program across the whole of the 
health care system. Accordingly, the Senate passed 
the Patient Safety and Quality Improvement Act of 
2005 (introduced by Senator James Jeffords, Ver- 
mont), which was subsequently passed by the 
House of Representatives and signed into law by the 
president. The law encourages health care providers 
to report errors and near misses to patient safety or- 
ganizations, defined as private or public organiza- 
tions that analyze reported patient safety data and 
develop strategic feedback and guidelines to 
providers on how to improve patient safety and the 
quality of care. The bill also includes protections for 
reporter and patient confidentiality, does not man- 
date a punitive reporting system, and does not alter 
existing rights or remedies available to injured pa- 
tients. This development offers enormous promise 
for systematic study and systemic interventions on 
behalf of patient safety, starting with reporting tools. 



Development of a Reporting Tool 

A systems-based approach to patient safety can 
adapt and integrate several human factors frame- 
works, including the following: 



• The SHEL (software/hardware/environment/ 
liveware) model. First introduced by Edwards 
(1972) and further developed by Hawkins 
(1987), this model is an organizational tool 
used to identify factors and work system ele- 
ments that can influence human performance 
(Molloy & O'Boyle, 2005). 

• The accident causation and generic error- 
modeling system framework. Developed by 
Reason (1990), this guiding tool constructs a 
sequence of events and circumstances that can 
help identify the unsafe conditions, actions, 
and decisions that led to a given incident. 

• The taxonomy of error (Hawkins, 1987; Ras- 
mussen, 1987) links skill-, rule-, and 
knowledge-based error types to appropriate 
levels of intervention. 

After reconciling the legal issues of a volun- 
tary reporting tool, the HISRS framework outlines 
the collection and analysis of data on errors and 
the tracking of the results of interventions (see 
figure 23.4). The reporting addresses multiple 
issues: 

• Information about the reporter. 

• In what medical sector are you involved? 

• What is your specific job (e.g., physician, ad- 
ministrator, nurse, laboratory technician)? 

• In what type of facility do you typically work 
(e.g., clinic, intensive care unit, pharmacy, out- 
patient clinic)? 

• Type of incident or hazard (e.g., wrong proce- 
dure or medicine; fall) 

• When did the incident/hazard occur? 

• How were you involved in the incident (or in 
discovering the hazard)? 

• Describe the working conditions. 

• Where did the incident happen? 

• What specific type of care was being rendered 
at the time of the incident or hazard? 

• Describe what happened. 

• What do you think caused the incident? Con- 
sider decisions, actions, inactions, information 
overload, communication, fatigue, drugs or al- 
cohol, physical or mental condition, proce- 
dures, policies, design of equipment, facility, 
workers (experience, staffing), equipment fail- 
ure (reasons), maintenance. 

• What went right? How was an accident 
avoided? Consider corrective actions, contin- 
gency plans, emergency procedures, luck. 



1) Event 



Who? 



What? 



When? 



• Where? 



Why? 



How? 



^> 



2) Reporter 



Ability to comprehend, analyze and deconstruct events 

Memory/accuracy 

Verbal ability 



How much does s/he know and not know? 
Willingness to divulge (fear of repercussion for self 
or colleagues) 



^> 



3) Form Completion 



Form^ 



Format 



• Items 



Detail 



• Ease of use 



^> 



4) Form Review/Data Entry 


Database Structure 

• Sufficient detail? 

• Well-defined? 


Analyst 

• Subject matter knowledge (medical and HF) 

• Ability to logically analyze 

• Willingness to return call and consult 

• Validity/reliability 



^ 



5) Data Analyses 


Database 

• Does taxonomy support analysis? 

• Search and sort algorithms 


Analyst 

• Respect for limitations of the data, database 

• Knowledge of specific definitions of data items 

• Creativity in search techniques 

• Willingness to perform complete, accurate analysis 



o 



6) HISRS Products 



1 Safety product objectives 
1 Target audience 



• Product content 
(sufficient detail? new info?) 



• Product format 

(easy to use and find info?) 

• Web site database 



O 



7) Track Responses by Industry, Agencies, MDs 



Tracking Results > 



• Direct tracking: response to alert bulletins — what was done? 

• Indirect tracking: trends — hazard reports drop? 

• Industry survey: Who uses it? Is it beneficial? 



Figure 23.4. Health care information safety report system (HISRS) framework. 
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How can we prevent similar incidents (correct 
the hazard)? What changes need to be made? 
By whom? Describe lessons learned, safety 
tips, and suggestions. 



It remains unclear how these taxonomies of error 
map onto specific cognitive deficits (e.g., of atten- 
tion, perception, memory; see figure 23.3) that 
lead to errors. 



Taxonomy of Error 

To effectively catalog errors for analysis, a taxon- 
omy of information-processing categories is needed. 
An example of one system of taxonomy is summa- 
rized below (cf. Norman, 1981, 1988; Reason, 
1984, 1990). 

Knowledge-based mistakes signify inappropriate 
planning due to failure to comprehend because the 
operator is overwhelmed by the complexity of a 
situation and lacks information to interpret it cor- 
rectly. For example, a specialist may misdiagnose a 
patient because a primary care provider omitted 
relevant information when referring the patient. 

Rule-based mistakes occur when operators be- 
lieve they understand a situation and formulate a 
plan by if-then rules, but the if conditions are not 
met, a "bad" rule is applied, or the then part of the 
rule is poorly chosen. For example, a doctor mis- 
diagnoses a patient with an extremely rare disor- 
der because of similarity of symptoms, when the 
diagnosis of a more common disorder is more 
probable. 

Slips are errors in which an intention is incor- 
rectly carried out because the intended action se- 
quence departs slightly from routine, closely 
resembles an inappropriate but more frequent ac- 
tion, or is relatively automated. The "reins of ac- 
tion" or perception are captured by a contextually 
appropriate strong habit due to lack of close moni- 
toring by attention. 

Lapses represent failure to carry out an action 
(omission of a correct action rather than commis- 
sion of an incorrect action), may be caused by 
interruption of an ongoing sequence by another 
task, and give the appearance of forgetfulness. For 
example, a doctor may intend to write a prescrip- 
tion for a patient in intensive care, but forgets the 
prescription when his attention is taken away by 
an emergency situation. 

The promise of error taxonomies is to provide 
an organizing framework for identifying common 
causes and mitigation strategies from seemingly 
unrelated instances. Despite their promise, tradi- 
tional error taxonomies have been generally in- 
effective in generating useful mitigation strategies. 



Systemic Factors 

A more constructive approach to classifying errors 
may be to focus on collecting narrative descrip- 
tions of events and then using a multidisciplinary 
team to identify the factors that contributed to the 
mishap. Under this framework, errors may then 
by classify by level of the health care system and 
areas of cognition that contributed to the error. 
An important reason for this is that the factors 
that shape the outcome of any particular situation 
have their roots in several levels of system descrip- 
tion (figure 23.5; Rasmussen et al., 1994). A level 
of description that focuses on the information- 
processing limits of cognition may fail to identify 
the contribution of management or team deficien- 
cies. A multidisciplinary approach brings multiple 
viewpoints to the analysis of an error and often 
identifies a range of contributing factors and 
system deficiencies that lead to the error. Figure 
23.5 shows the range of potential viewpoints that 
a multidisciplinary team can adopt in under- 
standing the range of factors that contribute to er- 
rors and then identifying successful mitigation 
strategies. 

A notable example of a systemic approach to 
mitigating medical errors is the cognitive work 
analysis (CWA) framework (Rasmussen et al., 
1994; Vicente, 1999). CWA provides a multilevel 
taxonomy to classify and analyze medical errors, 
defining several potential levels for neuroer- 
gonomic intervention. CWA comprises five layers, 
each analyzing a different aspect of an application 
domain (see figure 23.6). The first layer of con- 
straint is the work domain, a map of the environ- 
ment to be acted upon. The second layer is the set 
of control tasks that represents what needs to done 
to the work domain. The third layer is the set of 
strategies that represents the various processes by 
which action can be effectively carried out. The 
fourth layer is the social-organizational structure 
that represents how the preceding set of demands 
is allocated among actors, as well as how those ac- 
tors can productively organize and coordinate 
themselves. Finally, the fifth layer of constraint is 
the set of worker competencies that represent the 



The actual 
work environment 




Removing action alternatives: 
Defining behavior-shaping 
constraints at progressively 
narrower envelopes 



Figure 23.5. Shows a range of perspectives a multidisciplinary approach can adopt in a systematic 
analysis of errors to identify effective mitigation strategies. From Rasmussen et al. (1994). 



Figure 23.6. The cognitive work 
analysis framework is an example of a 
constraint-based approach compris- 
ing five layers: work domain, control 
tasks, strategies, social-organizational 
(soc-org) analysis, and worker com- 
petencies. These relationships are log- 
ically nested with a progressive 
reduction of degrees of freedom. 
Adapted from Vicente (1999). 
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Table 23.1. Relationships Between the Five Phases of Cognitive Work 
Analysis and Various Classes of System Design Interventions 



Phase 



Systems Design Intervention 



Work Domain 

What information should be measured? 
What information should be derived? 
How should information be organized? 

Control Tasks 

What goals must be pursued and what 
are the constraints on those goals? 
What information and relations are 
relevant for particular classes of situations? 

Strategies 

What frames of reference are useful? 
What control mechanisms are useful? 

Social-Organizational 

What are the responsibilities of all of 

the actors? 

How should actors communicate with 

each other? 

Worker Competencies 

What knowledge, rules, and skills do 

workers need to have? 



Sensors 
Models 
Database 

Procedures or automation 
Context-sensitive interface 



Dialogue modes 
Process flow 



Role allocation 
Organizational structure 



Selection, training, and 
interface form 



Source: Vicente (1999). 



capabilities required for success. Given this breadth, 
CWA provides an integrated basis for the develop- 
ment of mitigation strategies for medical error in 
general and neurological misdiagnosis in particu- 
lar. CWA, described below, has already been suc- 
cessfully applied to aviation safety. 

Figure 23.6 illustrates how these five layers of 
constraint are nested. The size of each set in this 
diagram represents the productive degrees of free- 
dom for actors, so large sets represent many rele- 
vant possibilities for action, whereas small sets 
represent fewer relevant possibilities for action. 

The CWA framework also comprises modeling 
tools that can be used to identify each layer of con- 
straint (Rasmussen et al., 1994; Vicente, 1999). 
Table 23.1 shows how each of these models is 
linked to a particular class of design interventions. 
The list is illustrative, not definitive or exhaustive, 
but it shows how CWA is geared toward uncover- 
ing implications for systems design that are clearly 
relevant to neuroergonomic interventions. Begin- 
ning with the work domain, analyzing the system 



being controlled provides insight into what infor- 
mation is required to understand its state. In turn, 
this analysis has implications for the design of sen- 
sors and models. The work domain analysis also 
reveals the functional structure of the system being 
controlled. These insights can then be used to de- 
sign a database that keeps track of the relationships 
between variables, providing a coherent, integrated, 
and global representation of the information con- 
tained therein. CWA has been successfully applied 
to aviation and process control. The underpinnings 
of a single error may have complex roots and may 
arise from a concatenation of problems across sev- 
eral of the layers. 



Safety Management Through Error 
Reports and Proactive Analysis: 
A Control Theoretical Approach 

Control theory provides a useful framework for 
describing some of the fundamental challenges to 
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reducing the incidence of medical errors and en- 
hancing patient safety. Control theory is a systems 
analysis concept that describes the relationship be- 
tween the current system state, the goal state, envi- 
ronmental disturbances, operating characteristics 
of the system components, and the control strate- 
gies used to achieve the goal state (Jagacinski & 
Flach, 2002). 

CWA evaluates system characteristics to iden- 
tify organizational and cognitive stress that may in- 
duce errors before errors occur. Error reporting 
and analysis examines the causes of errors after 
they occur. Both of these approaches generate miti- 
gation strategies that can be introduced into the 
system to reduce error rates in the future. In the 
language of control theory, these approaches are 
examples of feedback and feedforward control. 

Feedback control is analogous to safety man- 
agement through error reporting systems. With 
feedback control, mitigation strategies are adjusted 
based on observed differences between the goal 
and observed levels of safety. This approach re- 
quires a comprehensive reporting of errors and in- 
troduces a lag between when errors occur and 
when mitigation strategies can be deployed. 

Feedforward control is analogous to safety 
management through cognitive work analysis. This 
approach requires an ability to fully describe the 
medical diagnosis system and catalog all the factors 
that influence errors. 

Feedforward and feedback control have well- 
known capabilities and limits that can help clarify 
the requirements for the study of medical error. 
Specifically, unmeasurable errors and the time lag 
between error reporting and intervention design 
provide a rationale for feedforward control. No 
error-reporting system can capture every important 
error in a timely manner. Likewise, the rationale for 
feedback control comes from the difficulty in com- 
prehensively describing the cognitive and organiza- 
tional stressors of a complex system such as the one 
that supports neurological diagnosis. No cognitive 
work analysis will identify all possible error mecha- 
nisms. The feedforward approach (cognitive work 
analysis) and the feedback approach (error reporting 
and analysis) are complementary strategies that are 
both required to mitigate medical error. 

The control theoretical framework integrates 
feedback and feedforward approaches to safety 
management and identifies several other critical re- 



quirements associated with the process of identify- 
ing and evaluating mitigation strategies: 

• What cannot be measured cannot be controlled. 
Safety measurement is stressed, with effective 
strategies for capturing incident data. 

• Mitigation, not blame. Reporting mechanisms 
are defined that go beyond administrative and 
punitive purposes to address the underlying 
factors that contributed to the failure. Error re- 
ports focus on identifying mitigation strate- 
gies, not assigning blame. 

• Getting the word out. Attention is focused on 
the need to identify information pathways into 
which the understanding of failures can be fed 
so that the lessons learned from errors can be 
widely disseminated and used to develop ef- 
fective technological and social interventions. 

• Ongoing evaluation of strategies. Mitigation 
strategies are continuously evaluated using 
contemporary error data. 

Figure 23.7 summarizes how this approach in- 
tegrates the two complementary strategies described 
in this chapter: a proactive strategy of cognitive 
work analysis and a reactive strategy of error report- 
ing and analysis (Lee & Vicente, 1998). 



Examples of Neuroergonomic 
Interventions 

Having reviewed a framework for understanding 
human error, taxonomies of error, reporting sys- 
tems, feedback loops, CWA, and cultural and legal 
factors surrounding error reporting and patient 
safety, we now describe a few incipient interven- 
tions. These current efforts to advance patient 
safety aim to improve interactions between health 
care personnel, systems, products, environments, 
and tools. These interventions can involve proce- 
dural interventions and policy changes, as outlined 
in National Patient Safety Goals (Joint Commis- 
sion on Accreditation of Healthcare Organizations, 
2005). 

A simple example of safety-relevant culture 
change is limiting the work week of postgraduate 
physicians-in-training to 80 hours. This interven- 
tion by the American College of Graduate Medical 
Education aimed to (1) curtail the traditional 
Oslerian abuse of physicians in training, also known 
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Figure 23.7. A control theoretical framework for safety management in medical 
systems. 



as residents or house staff (perhaps because many 
of these doctors were always in the house); and 
(2) improve patient safety by reducing cognitive 
errors due to physician fatigue (see also chapter 14, 
this volume). 

Neuroergonomic interventions may be initi- 
ated at different levels in the CWA framework out- 
lined above, using techniques described 
throughout parts II— VI of this book (including 
neuroergonomics methods, perception and cogni- 
tion, stress, fatigue and physical work, technology 
applications, and special populations). In general, 
these interventions are in early phases of planning 
and development but hold substantial promise for 
improving patient safety. Relevant efforts can in- 
volve monitoring of individual health care 
providers, patients, and processes, and systems for 
tracking the ongoing performance of health care 
workers, teams, and patients (see chapter 8, this 
volume). These efforts can take advantage of 
miniature physiological sensors and monitoring 
devices, improved health care information dis- 
plays, systems for enhanced communications be- 
tween individuals, offices, and institutions, and 
applications from virtual reality (VR). 

As outlined earlier, health care areas and pro- 
cesses to assess errors and improve safety are ideally 
informed by reporting systems that indicate points 
or levels in the system where problems are local- 
ized. Consequently, new opportunities for interven- 
tion arise with the passage of the Patient Safety and 
Quality Improvement Act of 2005, allowing for 



reporting systems with safe harbors and patient 
confidentiality protections in the context of patient 
safety organizations. This new legislation should 
allow more comprehensive reporting of adverse 
events that allow CWA analyses. 

Accordingly, the chapter describes an interface 
design system that uses CWA, an HISRS, and 
principles of ecological interface design to miti- 
gate diagnostic errors in long-loop feedback sys- 
tems. The chapter also reviews techniques for 
tracking the patient through the health care sys- 
tem using bar codes, applications of augmented 
reality in continuous control tasks with short-loop 
feedback to improve the safety of surgical and en- 
doscopic procedures, and telepresense setups, al- 
lowing the mind of the expert to operate at a 
distance. 

Ecological Interface Design: Mapping 
Information Requirements onto 
a Computer Interface to Reduce 
Diagnosis Error 

Ecological interface design (EID) is a systems de- 
sign framework that complements the systems 
analysis framework comprising CWA. EID takes 
the information requirements identified by CWA 
models and turns them into specific interface de- 
signs. Specific design principles enable this pro- 
cess to present information in a format that makes 
it easier for people to understand what they need 
to get a job done (Vicente & Rasmussen, 1992). 
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Providing rich feedback in the interface has the 
potential to minimize errors and facilitate error de- 
tection and, thus, error recovery. 

The EID framework has been applied to com- 
plex systems, including aviation (Dinadis & Vi- 
cente, 1999), computer network management 
(Kuo & Burns, 2000), fossil-fuel power plants 
(Burns, 2000), information retrieval (Xu, Dainoff, 
& Mark, 1999), military command and control 
(Burns et al., 2000), nuclear power plants (Itoh, 
Sakuma, & Monta, 1995), petrochemical plants 
(Jamieson & Vicente, in press), and software engi- 
neering (Leveson, 2000). Applications of EID to 
medicine promise to improve patient safety in a va- 
riety of settings including intensive care (e.g., Haj- 
dukiewicz, Vicente, Doyle, Milgram, & Burns, 
2001; Sharp & Helmicki, 1998). An EID-based 
computer application could also be developed to 
mitigate diagnosis error in long-loop feedback set- 
tings relevant to community and office-based clini- 
cal medical practice settings. The computer 
application could organize information about each 
case in a graphical format that combines the his- 
tory and results of previous medical tests into an 
easily accessible format. 

The evaluation of the computer application 
could include two approaches. First, the applica- 
tion can be evaluated in a controlled setting using a 
selection of difficult diagnostic cases culled from 
reported errors. These cases could be reconstructed 
and presented to physicians with and without the 
EID-based computer application. Second, the com- 
puter application could be disseminated to the 
physician community, where it would be evaluated 
in terms of comments from the physicians and in 
terms of reductions in errors measured by the error 
reporting system. 

Controlled experiments and a reliable error re- 
porting system allow measurements of the benefits 
of such a tool in terms of error reduction, com- 
pared to base rates of error in current practice. 
Such EID-based computer application could en- 
hance diagnostic accuracy and lead physicians to 
be more confident about correct diagnoses and 
less confident about incorrect diagnoses, com- 
pared to the physicians using traditional organi- 
zational schemes. Also, physicians using an 
EID-based computer application would conduct a 
more thorough diagnosis and examine a broader 
range of information, compared to the traditional 
approach. 



Tracking the Patient Through the System 
Using Bar Codes 

Bar codes are commonly used in supermarkets to 
improve efficiency and reduce charge errors at 
checkout counters. They can also be used to im- 
prove patient safety and avoid a variety of error 
types. For instance, heath care personnel can scan 
information into a computer terminal at the point 
of care from bar codes on their ID badges, a pa- 
tient's ID bracelet, and medicine vials. The com- 
puter can track the patient through the health care 
system and improve situation awareness for the in- 
dividual caregiver and health care team regarding 
where a patient is, what procedures are being 
done, and by whom (Yang, Brown, Trohimovich, 
Dana, & Kelly, 2002). Bar code-enabled point-of- 
care software can mitigate transfusion and tissue 
specimen identification errors (Bar Code Label for 
Human Drug Products and Blood, 2003) and 
make certain a patient receives the proper proce- 
dure or drug dose, on time, while warning against 
potential adverse drug interactions or allergic re- 
actions (Patterson, Cook, & Render, 2002). A 
"smart" drug infusion pump can read drug iden- 
tity, concentration, and units from the bar code on 
a drug vial, display these values, and remind the 
operator to check them against the patient's pre- 
scription (Tourville, 2003). It could prevent harm 
from drug dosing errors by halting drug adminis- 
tration and issuing a warning if it cannot read the 
bar code on the vial, if the drug dose is out of 
range, or if it detects a potentially harmful interac- 
tion with another drug the patient is taking. Simi- 
lar technology may help patients avoid errors in 
self-administration of rapidly acting and poten- 
tially dangerous drugs, as in diabetic patients who 
self-inject insulin to control their blood sugar lev- 
els. Such technology can even be combined with a 
small global positioning system transmitter that 
can be attached to a patient at risk for getting lost. 
Such an intervention might help rescue a nursing 
facility resident with a memory disorder such as 
Alzheimer's disease who wanders outdoors in thin 
clothes in midwinter, unattended and unnoticed. 
Further in the future, galvanic vestibular stimula- 
tion (Fitzpatrick, Wardman, & Taylor, 1999) might 
be used to modulate patient posture and gait 
(Bent, Inglis, & McFadyen, 2004), perhaps even 
to steer lost patients back or prevent unsteady pa- 
tients from falling. 
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Augmented Reality and Virtual Reality 

VR environments can be used to train novice per- 
sonnel in safety critical care procedures on virtual 
patients without risk of harm to a patient. Aug- 
mented reality (AR) and augmented cognition ap- 
plications can enhance the planning, conduct, and 
safety of complex surgeries by skilled surgeons. It 
is possible to enhance the display of information 
to improve situation awareness by a health care 
worker and team (see Ecological Interface Design). 
Continuously monitored indices of performance 
and arousal of the worker can inform personnel 
and their supervisors of impairments or perfor- 
mance decline to avert impending errors and in- 
juries, using optimized alerting and warning 
signals. These preventive strategies dovetail with 
applications of crew resource management (CRM) 
training techniques (Wiener, Kanki, & Helmreich, 
1995), first developed to prevent errors in aviation 
settings where stress levels and workload are high 
and communications may fail, leading to disaster. 
CRM may be applied to mitigate errors in emer- 
gency rooms and other stressful critical care set- 
tings by improving communications and situation 
awareness among the health care team. 

VR also provides neuroergonomic tools to aid 
an operator's spatial awareness and performance in 
complex navigational situations, such as perform- 
ing complex surgeries in small spaces and recog- 
nizing and coping with ramifications of anatomical 
variations (McGovern & McGovern, 1994; Satava, 
1993). These systems can use principles from AR, 
discussed in chapter 17 in terms of combining of 
real and artificial stimuli, with the aim of improv- 
ing human performance and insight. This gener- 
ally involves overlaying computer-generated 
graphics on a stream of video images so that these 
virtual images appear to be embedded in the real 
world. 

For example, an AR application — projecting 3- 
D radiographic information on the location of a 
patient's organs on surface of a patient's own 
body — would allow a physician to see through a 
patient to localize and better understand and treat 
disease processes. These systems ultimately should 
reduce spatial errors and navigation problems (see 
chapter 9, this volume) that may result in 
procedure-related iatrogenic injuries to patients. 

Imagine a system that uses commercially 
available video products and computer hardware 



to combine video of an actual patient with 3D 
computed tomography (CT) scan or magnetic res- 
onance (MR) images of the brain to help in plan- 
ning surgical operations. The surgeon can view 
the CT or MR images projected over images of the 
actual patient. These methods can be applied be- 
fore and during surgery to locate borders of a 
lesion. For example, using these techniques, a 
neurosurgeon could plan the best site for a skin 
incision, craniotomy, and brain incision, practice 
the procedure, and limit injury to normal brain 
tissue (Spicer & Apuzzo, 2003). 

In a similar vein, Noser, Stern, and Stucki 
(2004) assessed a synthetic vision-based applica- 
tion that was designed to find paths through com- 
plex virtual anatomical models of the type 
encountered during endoscopy of colons, aortas, 
or cerebral blood vessels (Gallagher & Cates, 
2004). These spaces typically contain loops, bottle- 
necks, cul de sacs, and related impasses that can 
create substantial challenges, even for highly 
skilled operators. The application found collision- 
free paths from a starting position to an end-point 
target in a timely fashion. The results showed the 
feasibility of automatic path searching for interac- 
tive navigation support, capable of mitigating 
injuries to anatomical structures caused by naviga- 
tion errors during endoscopic procedures. 

A navigational aid might display, in real time, 
the shape of a flexible endoscope inside the colon 
(Cao, 2003). Spatial orientation error and work- 
load may be reduced in operators using such a 
device, which provides additional short-loop feed- 
back to the operator for error correction. Similar 
VR-based strategies can be applied during teach- 
ing, training (Seymour et al., 2002), practicing, 
and actual surgery to reduce errors in sinus sur- 
gery (Satava & Fried, 2002), urological surgery, 
neurosurgery (Peters, 2000), and breast biopsies. 

To see inside patients and guide physicians 
during internal procedures, neuroergonomists are 
testing augmented reality systems that combine ul- 
trasound echography imaging, laparoscopic range 
imaging, a video see-through head-mounted dis- 
play, and a graphics computer with the live video 
image of a patient (Rolland & Fuchs, 2000). Figure 
23.8 shows the AR setup being applied to im- 
prove performance and reduce errors during 
ultrasound-guided breast biopsy. This procedure 
requires guiding a biopsy needle to a breast lesion 
with the aid of ultrasound imagery, requires good 
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Figure 23.8. Shows the augmented reality setup being applied to improve performance and re- 
duce errors during ultrasound-guided breast biopsy. This procedure requires guiding a biopsy nee- 
dle to a breast lesion with the aid of ultrasound imagery, requires good 3-D visualization skills and 
hand-eye coordination, and is difficult to learn and perform. 



3-D visualization skills and hand-eye coordination, 
and is difficult to learn and perform. Figure 23.9 
shows an AR image for localizing a lesion during 
sinus surgery. 

Action at a Distance: Extending 
the Human Mind and 
Distributing Expertise 

Neuroergonomics applications can not only help to 
guide and focus an operators mind but can also 
empower the operator to act at a distance. Prelimi- 
nary research suggests the feasibility of telepresence 
operations, allowing a remotely stationed operator 
to perform invasive procedures on a patient located 
far beyond arm's length in a different room. Think 
in terms of an expert using a robotic arm manipu- 
lating control of the nuclear fuel rods in a nuclear 
power plant, the Canadian payload arm of the 
Space Shuttle, the arms of an unmanned subma- 
rine, or a digging tool of the Mars Rover. In the neu- 
roergonomic health care application, the patient is 
located in a remote procedure room containing a 
stereoscopic camera for 3-D visualization of the 



operative field and a robot with tools to implement 
the operator's commands to perform medical pro- 
cedures. Such a setup uses some of the same tech- 
nology that allows the control of unmanned 
robotic vehicles used in surveillance and in under- 
sea exploration and space exploration. It would 
extend the human mind, allowing action at a dis- 
tance in remote locations around the globe for 
treatment of patients where there is insufficient lo- 
cal expertise, such as in rural areas, battlefields, 
ships, and perhaps even in extraterrestrial settings 
aboard space ships, space stations, and in space 
colonies. 



Conclusion 

Medical errors depend on multilevel factors from 
individual performance to systems engineering. 
Overwork, understaffing, and overly complex pro- 
cedures may stress the mental resources of health 
care practitioners. Available information from ex- 
isting sources (such as closed malpractice claims 
analyses of visible tip-of-the-iceberg safety errors) 
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Figure 23.9. Shows an augmented reality image for a localized lesion during sinus surgery 



indicates that errors of diagnosis are an important 
type of medical error. Taxonomies of error provide 
a heuristic framework for understanding such er- 
rors but have not yet led to concrete improvements 
in safety Feedback to the operator is essential to 
mitigation of errors in health care systems. In med- 
ical subspecialties, these feedback loops can be 
long and indirect. Errors of diagnosis can be re- 
duced by redesigning procedures and systems, us- 
ing techniques borrowed from other safety-critical 
industries. 

New opportunities arise for reducing errors 
and improving patient safety following the passage 
from bill to law of the Patient Safety and Quality 
Improvement Act of 2005. This development al- 
lows for voluntary reporting systems with safe har- 
bors and patient confidentiality protections. It 
permits the development of more comprehensive 
reporting systems amenable to cognitive work 
analysis for localizing and mitigating errors and 
improving medical care at all levels of the health 
care industry. These interventions can use modern 



tools and techniques including ecological interface 
design, information display technologies, virtual 
reality environments, and telepresence systems 
that extend the mind to distribute expertise and 
allow skilled medical action and procedures at a 
distance. 



Main Points 

1. Medical errors depend on multilevel factors 
from individual performance to systems 
engineering. 

2. Overwork, understaffmg, and complex 
procedures may stress the mental resources of 
health care personnel and increase the 
likelihood of error. 

3. Errors can be reduced by redesigning 
procedures and systems, using techniques 
borrowed from other safety-critical industries 
such as air transportation and nuclear 
power. 
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4. Feedback to the operator is essential to 
reducing errors in health care systems. 
Feedback loops can be short and direct in 
surgical specialties, or long and indirect in 
medical specialties. 

5. These interventions can use modern tools and 
techniques including ecological interface 
design, information display technologies, and 
virtual reality applications. 
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Future Prospects for 
Neuroergonomics 



The preceding chapters present strong evidence for 
the growth and development of neuroergonomics 
since its inception a few years ago (Parasuraman, 
2003). The ever-increasing understanding of the 
brain and behavior at work in the real world, the de- 
velopment of theoretical underpinnings, and the re- 
lentless spread of facilitative technology in the West 
and abroad are inexorably broadening the substrates 
for this interdisciplinary area of research and prac- 
tice. Neuroergonomics blends neuroscience and er- 
gonomics to the mutual benefit of both fields and 
extends the study of brain structure and function 
beyond the contrived laboratory settings often used 
in neuropsychological, psychophysical, cognitive 
science, and other neuroscience-related fields. 

Neuroergonomics is providing rich observa- 
tions of the brain and behavior at work, at home, 
in transportation, and in other everyday environ- 
ments in human operators who see, hear, feel, at- 
tend, remember, decide, plan, act, move, or 
manipulate objects among other people and tech- 
nology in diverse, real-world settings. The neuroer- 
gonomics approach is allowing researchers to ask 
different questions and develop new explanatory 
frameworks about humans at work in the real 
world and in relation to modern automated sys- 
tems and machines, drawing from principles of 



neuropsychology, psychophysics, neurophysiology, 
and anatomy at neuronal and systems levels. 

Better understanding of brain function, as out- 
lined in the chapters on perception, cognition, and 
emotion, is leading to the development and refine- 
ment of theory in neuroergonomics, which in turn 
is promoting new insights, hypotheses, and re- 
search. For example, research on how the brain 
processes visual, auditory, and tactile information is 
providing important guidelines and constraints for 
theories of information presentation and task de- 
sign, optimization of alerting and warning signals, 
development of neural prostheses, mitigation of er- 
rors by operators whose physiological profiles indi- 
cate poor functioning or fatigue, and development 
of robots that emulate or are part of human beings. 

Specific challenges for this new field, both in 
the near term and beyond, are outlined in each 
part of this book (part II, Neuroergonomics Meth- 
ods; part III, Perception, Cognition, and Emotion; 
part iy Stress, Fatigue, and Physical Work; part V, 
Technology Applications; and part VI, Special Pop- 
ulations). In this closing chapter, we briefly discuss 
prospects for the future, both in the near term and 
the longer term. We also address some of the gen- 
eral challenges facing neuroergonomics research 
and practice. These include issues of privacy and 
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ethics and the development of standards and guide- 
lines to ensure the quality and safety of a host of 
new procedures and applications. 



The Near Future 

Simulation and Virtual Reality 

An imminent challenge in neuroergonomics will 
be to disseminate and advance new methods for 
measuring human performance and physiology in 
natural and naturalistic settings. It is likely that 
functional magnetic resonance imaging (fMRI) 
methods will be further applied to study brain activ- 
ity in tasks that simulate instrumental activities of 
daily living, within the constraints of the scanner en- 
vironment. The ability to image brain activity during 
complex, dynamic behavior such as driving and 
navigation tasks will enhance our understanding of 
the neural correlates of complex behaviors and the 
performances of neurologically normal and im- 
paired individuals in the real world. Further use of 
fMRI paradigms involving naturalistic behaviors will 
depend on improved software design and analytic 
approaches such as independent component analy- 
sis to decompose and understand the complex data 
sets collected from people engaged in these complex 
tasks. This exciting future also depends on advances 
in brain imaging hardware, fMRI experimental de- 
sign and data analyses, ever-better virtual reality 
(VR) environments, and stronger evidence to deter- 
mine the extent to which tasks in these environ- 
ments are valid surrogates for tasks in the real world. 
Multidisciplinary vision and collaboration have 
established VR as a powerful tool to advance 
neuroergonomics. VR applications — using systems 
that vary from surrealistic, PC-based gaming plat- 
forms to high-fidelity motion-based simulation and 
fully immersive, head-mounted VR — are providing 
computer-based synthetic environments to train, 
treat, and augment human behavior and cognitive 
performance in renditions of the real world. An 
unanswered question regarding the fidelity of us- 
able VR environments is, "How low can you go?" A 
range of VR systems will continue to be used to in- 
vestigate the relationships between perception, ac- 
tion, and consciousness, including how we become 
immersed in an environment and engaged by a task, 
and why we believe we are where we think we are. 



Augmented reality (AR) systems combine real 
and artificial stimuli, often by overlaying computer- 
generated graphics on a stream of video images, so 
that the virtual objects appear to be embedded in 
the real world. The augmentation can highlight 
important objects or regions and superimpose in- 
formative annotations to help operators accomplish 
difficult tasks. These systems may help aircraft pi- 
lots maintain situational awareness of weather, air 
traffic, aircraft state, and tactical operations by us- 
ing a head-mounted display that displays such in- 
formation and enhances occluded features (such as 
the horizon or runway markings); other systems 
may allow a surgeon to visualize internal organs 
through a patients skin. Immediate challenges in 
creating effective AR systems include modeling the 
virtual objects to be embedded in the image, pre- 
cisely registering the real and virtual coordinate 
systems, generating images quickly enough to avoid 
any disconcerting lag when there is relative move- 
ment, and building portable devices that do not 
encumber the wearer. 

Physiological Monitoring 

Advances in physiological measurements will per- 
mit additional observations of brain function in 
simulated and real-world tasks. Electroencephalo- 
gram (EEG) data should provide further data about 
changes in regional functional brain systems acti- 
vated by ongoing task performance to complement 
the evidence from fMRI, positron emission tomog- 
raphy (PET), and other techniques. As discussed 
by Gevins and Smith (chapter 2), such studies are 
possible because EEG patterns change predictably 
with changes in task load, mental effort, arousal, 
and fatigue, and can be assessed using algorithms 
that combine parameters of the EEG power spectra 
into multivariate functions. Although these EEG 
data lack the 3-D spatial resolution of brain fMRI 
or PET, EEG is more readily applied in tasks that 
resemble those that an individual might encounter 
in a real- world environment. 

Event-related potentials (ERPs) provide addi- 
tional insights and applications for neuroergonom- 
ics and are computed by averaging EEG epochs 
time-locked to sensory, motor, and cognitive events. 
Although ERPs also have lower spatial resolution 
than fMRI, ERP resolution is improving because of 
new source localization techniques. Moreover, ERPs 
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have better temporal resolution for evaluat- 
ing neural activity than other neuroimaging tech- 
niques do. As Fu and Parasuraman (chapter 3) 
discuss, ERP components such as P300, Nl, PI, 
ERN, and LRP will provide additional information 
relevant to neuroergonomic assessments of mental 
workload, attention resource allocation, dual-task 
performance, error detection and prediction, and 
motor control. Advances in device miniaturization 
and portability will continue to enhance the utility 
of EEG and ERP systems. Development of auto- 
mated systems, in which human operators moni- 
tor control system functions, will provide further 
opportunities to use ERP-based neuroergonomic 
measures and theories relevant to brain function at 
work. 

Noninvasive optical imaging tools, such as 
near-infrared spectroscopy (NIRS), assess neuronal 
activity that occurs directly after stimulus presenta- 
tion or in preparation for responses and hemody- 
namic changes that occur a few seconds later, and 
these add to the neuroergonomics toolbox (which 
includes fMRI, PET, EEG, and ERP). As Gratton 
and Fabiani (chapter 5) discuss, optical imaging 
provides a good combination of spatial and tempo- 
ral resolution of brain activity, can be deployed in a 
range of environments and experimental condi- 
tions, and costs little compared to magnetoen- 
cephalography, PET, and fMRI. Because optical 
imaging systems are relatively portable, they may 
be applied more commonly to map the time course 
of brain function and hemodynamic responses. 
Further advances will need to overcome the rela- 
tively low signal-to-noise ratio that affects the faster 
neuronal signal and the limited penetration by the 
technique to within a few centimeters beneath the 
scalp, which precludes measurements of activity in 
deeper brain structures (subcortical and brain 
stem) in adults. 

Images of the brain at work will be comple- 
mented by transcranial Doppler sonography (TCD), 
which allows fast and mobile assessment of task- 
related brain activation and related effects of work- 
load, vigilance, and sustained attention. Tripp and 
Warm (chapter 6) show that, like NIRS-based mea- 
surement of blood oxygenation, TCD offers a non- 
invasive and relatively inexpensive way to "monitor 
the monitor." TCD may prove especially useful to 
assess task difficulty and task engagement and to 
determine when human operators need rest and 



whether they may benefit from adaptive automa- 
tion systems designed to flexibly allocate tasks be- 
tween the operators and computer systems to 
mitigate operator workload and fatigue. 

Eyelid closure measurements can be used to 
monitor fatigue and falling asleep on the job or at 
the wheel. McCarley and Kramer (chapter 7) show 
that eye movement assessments provide an addi- 
tional window on perception, cognition, and how 
we search for meaningful information in displays 
and real- world scenes. Advances in gaze- 
contingent control procedures will enable neuroer- 
gonomics researchers to infer capability, strategies, 
and perhaps even the intent of operators who are 
inspecting the panorama in a variety of complex 
simulated and real-world environments. Real-time 
measures of where a user is gazing are already be- 
ing used to develop attentive user interfaces, 
namely devices and computers that know where a 
person is looking and therefore do or do not inter- 
rupt the user accordingly (Vertegaal, 2002). Evi- 
dence from high-speed measurements of eye 
movements may also be combined with other mea- 
sures (such as EEG, ERP optical imaging, heart 
rate, NIRS) to enhance the design of display de- 
vices aboard aircraft, automobiles, and industrial 
systems and improve real-time workload and per- 
formance assessment algorithms. 

It is important to recognize what can and can- 
not be achieved specifically by physiological moni- 
toring in extralaboratory environments. We must 
also be careful not to overstate what neuroer- 
gonomics can achieve given the noisiness of most 
real- world settings. Some recent programmatic ef- 
forts have set very ambitious goals for real-time 
assessment of operator cognitive state using physi- 
ological measures (St. John, Kobus, Morrison, & 
Schmorrow, 2004). Meeting these goals will re- 
quire substantial effort focused on the elimination 
of potential artifacts that may mask the signal of in- 
terest, or, more seriously, lead to flawed assess- 
ments of operator state. Given the diversity and 
magnitude of artifacts that are likely to be encoun- 
tered in natural settings, obtained artifact-free 
recordings, particularly in real time, will pose a con- 
siderable technical challenge. Nevertheless, there 
have been a number of promising developments in 
the design of miniaturized recording equipment 
that can withstand the rigors of operational environ- 
ments. Automated artifact rejection procedures have 
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also been developed, which, if proven robust, 
would help considerably in meeting the real-time 
monitoring challenge (see Gevins et al., 1995; see 
also chapter 2, this volume). 

The Brain in the Wild 

In addition to physiological monitoring, neuroer- 
gonomic evaluations can also involve assessment of 
other aspects of human behavior in natural envi- 
ronments, what Rizzo and colleagues call the brain 
in the wild (chapter 8). New technologies are al- 
lowing the development of various implementa- 
tions of "people trackers" using combinations of 
accelero meters, global positioning systems, video, 
and other sensors (e.g., to measure gross and fine 
body movement patterns, skin temperature, eye 
movements, heart rate, etc.) to make naturalistic 
observations of human movement and behavior in 
the wild. As Rizzo and colleagues discuss, these 
trackers can advance the goal of examining perfor- 
mance, strategies, tactics, interactions, and errors 
in humans engaged in real-world tasks, drawing 
from established principles of ethology. Besides is- 
sues of device development and sensor choice and 
placement for various classes of devices (outside 
looking inside, inside looking inside, inside look- 
ing outside), taxonomies are needed for classifying 
likely behavior from sensor output. Different sen- 
sor array implementations will provide unique data 
on how the brain interacts with diverse environ- 
ments and systems, at work and at play, and in 
health, disease, or fatigue states. 

Such measurement techniques are likely to be 
important in the assessment of stress and fatigue at 
work. Fatigue on the job is common in our 24- 
hour-service society. Operational demands in 
round-the-clock industries such as health care and 
transportation and industries that require shift 
work inevitably cause fatigue from sleep loss and 
circadian displacement, which contributes to in- 
creased cognitive errors and risk of adverse events 
(such as medical errors and airplane crashes). Pre- 
dicting and mitigating the risks posed by physio- 
logically based variations in sleepiness and 
alertness at work are key functions of neuroer- 
gonomics, as discussed in chapter 14. Near-term 
advances will depend on more unobtrusive tools 
for detecting fatigue and a better understanding of 
the effects of sleep-wake cycles and circadian biol- 
ogy on human performance. 



Cross-Fertilization of Fields 

Evidence from many of the above-mentioned neu- 
roergonomic areas will continue to converge with 
evidence from other established fields such as 
animal physiology and human neuropsychology. 
De Pontbriand (2005) also envisaged cross- 
fertilization between neuroergonomics and rapidly 
growing fields such as biotechnology. Neuroer- 
gonomics will also continue to provide a unique 
source of evidence in its own right. For example, 
Maguire (chapter 9) explains how neuroergonom- 
ics can lead to a better understanding of plasticity 
and dynamics within the brain's navigation sys- 
tems; she foresees an increasingly fruitful exchange 
whereby technological and environmental im- 
provements will be driven by an informed under- 
standing of how the brain finds its way in the real 
world. 

In this spirit of interdisciplinary cross- 
fertilization, Grafman (chapter 1 1) outlines a repre- 
sentational framework for understanding executive 
functions that underpin work-related performance 
in the real world. This performance depends on key 
structures in human prefrontal cortex that mediate 
decision making and implementation, planning and 
judgment, social cognition, tactics, strategy, fore- 
sight, goal achievement, and risk evaluation. Fur- 
ther, Bechara recognizes that real-world decisions 
are critically affected by emotions, in a process that 
reconciles cold cognition in the cortex with pro- 
cesses in the more primitive limbic brain. Emotion 
systems are a key factor in interaction between en- 
vironmental conditions and human cognitive pro- 
cesses at work and elsewhere in the real world. 
These systems provide implicit or explicit signals 
that recruit cognitive processes that are adaptive 
and advantageous for survival. Understanding the 
neural underpinnings and regulation of emotions 
and feelings is crucial for many aspects of human 
behavior and their disorders, including perfor- 
mance at work, and can help provide a model for 
the design of new generations of computers that in- 
teract with humans, as in chapter 18. 



The Longer- Term Future 

Breazeal and Picard (chapter 18) explain how find- 
ings in neuroscience, cognitive science, and human 
behavior inspire the development of robots with 
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social-emotional skills. Currently in an early stage 
of development, these relational robots hold prom- 
ise in future applications in work productivity and 
education, where the human user may perform 
better in cooperation with the robot. For example, 
embedding affective technologies in learning inter- 
actions with automated systems (robotic learning 
companions) should reveal what affective states are 
most conducive to learning and how they vary 
with teaching styles, and this information will hone 
the robot's ability to learn from a person. Thus, hu- 
mans will learn from machines and vice versa, par- 
allel to the vision of Grafman (chapter 1 1) in which 
cognitive neuroscience applications will inform 
and improve training and evaluation methods cur- 
rently employed by human factors experts. 

Before they augmented our minds, machines 
amplified our muscles. Machines began relieving 
stress on human muscles as soon as humans dis- 
covered wedges, sledges, wheels, fulcrums, and 
pulleys. Hancock and Szalma (chapter 13) point 
out that machines will continue to replace human 
muscle power to minimize stress and fatigue as 
much as possible. In this futuristic vision of neu- 
roergonomics, human intentions, indexed by inter- 
pretable brain states, will connect directly to these 
machines to bring intention to action. The effector 
might be a body part-sized electromechanical pros- 
thesis such as an artificial limb or hand, or a sub- 
stantial machine such as an exoskeleton robot, 
easily capable of bone-crushing labor. Additional 
neuroergonomic tools will monitor operators, 
identify signs of impending performance failure, 
and adjust system behavior and workload to miti- 
gate stress. 

Neuroergonomic countermeasures to stress, fa- 
tigue, and sleepiness can include systems for adap- 
tive automation in which the user and the system 
can initiate changes in the level of automation, trig- 
gered in part by psychophysiological measures. 
Operators may come to think of these adaptive sys- 
tems more as coworkers (rather than tools, ma- 
chines, or computer programs) and even expect 
them to behave like humans. To design these adap- 
tive systems, developers will need better informa- 
tion about human-machine task sharing, methods 
for communicating goals and intentions, and as- 
sessment of operator states of mind, including trust 
of and annoyance with robotic systems. These 
adaptive automation systems will be advanta- 
geously applied in settings where safety concerns 



surround the operator, system, and recipient of 
system services. Other potential applications might 
include a personal assistant, butler, secretary, or re- 
ceptionist; an adaptive house; and systems aimed 
at training and skill development, rehabilitative 
therapy, and entertainment. 

As discussed by Scerbo, adaptive automation 
systems are challenged when the wishes of two or 
more of the operators have conflicting goals. Such 
conflicts may arise in health care and can be exam- 
ined at several levels using a cognitive work analysis 
framework in research aimed at improving medical 
quality and reducing errors. Addressing such com- 
plexities will depend on a better understanding of 
how multiple operators and their brains behave in 
social situations in differently organized work envi- 
ronments. 

Neural engineering is a dimension of neuroer- 
gonomics related to the establishment of direct in- 
teractions between the nervous system and artificial 
devices. In this arena, technology is applied to im- 
prove biological function while lessons from biology 
inform technology, with cross-fertilization between 
molecular biology, electrophysiology, mathematics, 
signal processing, physics, and psychology. Brain- 
machine and brain-computer interfaces (BCIs) can 
interact with telecommunication technologies for 
controlling remote devices or for transmitting and 
receiving information across large distances, en- 
abling operations at a distance. 

As Mussa-Ivaldi and colleagues (chapter 19) 
discuss, advances in neuroengineering will depend 
on better knowledge and models of brain func- 
tions, at all levels ranging from synapses to sys- 
tems, and on learning how to represent these 
models in terms of discrete or continuous vari- 
ables. In this vein, Poggel and colleagues (chapter 
21) address how the brain encodes visual informa- 
tion to learn how to "talk" to the brain and restore 
vision in patients with visual impairment due to 
retinal lesions. Meeting this goal will depend on a 
better understanding of plasticity, perception, low- 
level signal processing, and top-down influences 
such as attention. 

Creating a retinal implant, a visual cortical im- 
plant, or any other neural implant is a challenge for 
cognitive scientists, surgeons, electrical engineers, 
material scientists, biochemists, cell biologists, and 
computer scientists. For decades to come, scien- 
tists will be busy developing algorithms to emulate 
neural functions and control neuroprostheses; 



386 



Conclusion 



semiconductor chips to implement the algorithms; 
microelectrode arrays to match the organic cytoar- 
chitectonic scaffold of the nerves, spinal cord, or 
brain; and procedures to tune a device to a given 
patient. Ideally, human neural signals could be 
read remotely via transduction of electromechani- 
cal signals without surgically invading the body 
These human-machine interfaces could be used to 
control a variety of external devices to improve hu- 
man function and mobility 

Riener (chapter 22) reviews neurorehabilita- 
tion robotics and neuroprosthetics and shows how 
robots can be applied to support improved recov- 
ery in patients recovering from upper motor neu- 
ron and lower motor neuron lesions due to stroke, 
trauma, and other causes. Robots will make mo- 
tion therapy more efficient by increasing training 
duration and number of training sessions, while re- 
ducing the number of therapists required per pa- 
tient. Patient-cooperative control strategies have 
the potential to further increase the efficiency of 
robot-aided motion therapy. Neuroprosthetic func- 
tion will be improved by applying closed-loop con- 
trol components, computational models, and 
better BCIs. 

Better BCIs will allow a user to interact with the 
environment without muscular activity (such as 
hand, foot, or mouth movement) and will require 
specific mental activity and strategy to modify brain 
signals in a predictive way. BCIs can be useful in 
augmented cognition applications (as described 
above) and may also help patients with paralysis 
due to amyotrophic lateral sclerosis, spinal cord in- 
jury, or other conditions. Pfurtscheller and col- 
leagues (chapter 20) review BCIs that use EEG 
signals. The challenges are to record, analyze, and 
classify brain signals and transform them in real 
time into a control signal at the output of the BCI. 
This process is highly subject specific and requires 
rigorous training or learning sessions. 



Guidelines, Standards, and Policy 

Rapid development of devices, techniques, and 
applications in the field of neuroergonomics (neu- 
roengineering, neural prostheses, augmented cogni- 
tion, relational robots, and so on) have outpaced the 
development of guidelines, which are recommenda- 
tions of good practice that rely on their authors' 
credibility and reputation, and of standards, which 



are formal documents published by standards- 
making bodies that are developed through exten- 
sive consultation, consensus, and formal voting. 
Guidelines and standards help establish best prac- 
tices and provide a common frame of reference 
across new device designs over time to foster com- 
munication between different user groups, without 
hindering innovation. They allow reviewers to as- 
sess research quality, compare results across studies 
for quality and consistency, and identify inconsis- 
tencies and the need for further guidelines and 
standards. 

Making fair guidelines and standards involves 
science, logic, policy, politics, and diplomacy to 
overcome entrenched interests and opinions, and 
to avoid favoritism toward or undue influence by 
certain persons or groups. It may be easier to pro- 
pose standards than to apply them in practice, yet 
it is important to develop standards proactively be- 
fore they become externally imposed. Relevant to 
neuroergonomics, the Food and Drug Administra- 
tion is charged with administering standards for 
implantable brain devices, such as deep brain stim- 
ulators to mitigate Parkinson's disease, spinal cord 
stimulators for pain relief, cochlear implants for 
hearing, cardiac pacemakers to treat heart rhythm 
abnormalities, and vagal nerve stimulators to treat 
epilepsy or, lately, depression. 

The Centers for Medicare and Medicaid Services 
will weigh in on efficacy and standards when it is 
asked to reimburse providers for rendering services 
that may as yet lack sufficient evidence to support 
clinical use (such as VR for phobias). Professional 
groups such as the American Academy of Neurology 
(AAN), American Academy of Neurosurgery Ameri- 
can Psychological Association, the Special Interest 
Group on Computer-Human Interaction of the As- 
sociation for Computing Machinery, and the Human 
Factors and Ergonomics Society may also intervene 
to establish evidence-based quality standards when 
a practice, device, or treatment becomes profession- 
ally relevant and widespread. 

For example, the Quality Standard Subcom- 
mittee of the AAN and similar groups in other 
medical subspecialties write clinical practice guide- 
lines to assist their members in clinical decision 
making related to the prevention, diagnosis, prog- 
nosis, and treatment of medical disorders, which 
may come to include neuroergonomic applications 
or devices (such as neural implants to treat blindness 
or paralysis). The AAN guidelines make specific 
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practice recommendations based upon a rigorous 
review of all available scientific data. Key goals are 
to improve health outcomes, determine if practice 
follows current best evidence, identify research pri- 
orities based on gaps in the literature, promote effi- 
cient use of resources, and influence related public 
policy. Standards and guidelines in specific appli- 
cation areas discussed in this book, such as VR sys- 
tems, have been published (Stanney, 2002). For a 
more comprehensive examination of standards 
across all areas of human factors and ergonomics, 
see Karwowski (2006). 



Ethical Issues 

Issues of privacy have been at the forefront since 
the advent of neuroergonomics (Hancock & 
Szalma, 2003) and are likely to continue to be so in 
the future too. Workers may be helped by auto- 
mated systems that detect when fatigue, stress, or 
anxiety increase to levels that threaten performance 
and safety. But is there a dark side to such meth- 
ods? For example, could those who seem especially 
stress or anxiety prone based on highly variable 
physiological measures and inaccurate predictive 
models be unfairly excluded from new opportuni- 
ties or promotions? How will workers behave and 
what are their rights to privacy when it becomes 
possible to record seemingly everything they do, all 
the time, from multiple channels of data on brain 
and body states? The proliferation of embedded 
monitoring devices will expose events and behav- 
iors that were once hidden behind cultural or insti- 
tutional barriers. What could be done with these 
data beyond their intended purpose of enhancing 
safety, reducing stress, improving performance and 
health, and preventing injury? 

In a similar vein, surveillance countermeasures 
to perceived terrorist threats include software in- 
tended to predict the intent of individuals bent on 
mayhem based on body movements, fidgeting, facial 
expression, eye glances, and other physiological in- 
dices. In chapter 13, Hancock and Szalma emphasize 
that we must be wary when private thoughts, opin- 
ions, and ideas — regardless of whether we like them 
or not — are unfairly threatened by other individuals, 
corporations, politicians, or arms of the state. 

Remarkable ethical issues may unfold over the 
next century concerning the cooperative relationship 
between humans and machines at physical, mental, 



and social levels. How much should a person trust 
an automated adaptive assistant, avatar, virtual hu- 
man, or affective computer that is in the loop with 
the human operator and potentially serving as a 
counselor, physician, friend, coworker, or supervi- 
sor? Who is the boss? How shall we mitigate con- 
cerns of control over human minds, bodies, and 
institutions by implants, robots, and software? 

Modern discourse on somatic cell nuclear 
transfer (cloning) has sparked enormous hope and 
controversy at the borders between science, faith, 
policy, and politics. Emerging applications in neu- 
roergonomics that are capable of reading human 
brain activity, predicting human failures, creating 
emotive robots, virtual human coworkers, com- 
panions, and bosses — and of hybridizing human 
and machine — may face similar trials (see also 
Clark, 2003). Nevertheless, we must face these 
challenges and not bury our heads in the sand like 
ostriches and hope they go away. Unless we (scien- 
tists and engineers) ourselves consider the ethical 
and privacy questions, others outside of science 
will decide the issues for us. The developments 
in neuroergonomics may lead to extraordinary 
opportunities for improved human-machine and 
human-human interaction. Others have noted that 
such developments may well represent the next 
major step in human evolution (Clark, 2003; Han- 
cock & Szalma, 2003). Neuroergonomic technolo- 
gies should be developed to serve humans and to 
help them engage in enjoyable and purposeful ac- 
tivity to ensure a well-evolved future. 



Conclusion 

As an interdisciplinary endeavor, neuroergonomics 
will continue to benefit from and grow alongside 
aggressive developments in neuroscience, ergonom- 
ics, psychology, engineering, and other fields. This 
ongoing synthesis will significantly advance our un- 
derstanding of brain function underlying human 
performance of complex, real-world tasks. This 
knowledge can be put to use to design technologies 
for safer and more efficient operation in various 
work and home environments and in diverse popu- 
lations of users. The basic enterprise of human fac- 
tors and ergonomics — how humans design, interact 
with, and use technology — can be considerably en- 
riched if we also consider the human brain that 
makes such activities possible. As the chapters in this 
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volume show, there have already been considerable 
achievements in basic research and application in 
neuroergonomics. The future is likely to yield more 
such advances. 
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Glossary 



adaptable automation Systems in which changes 
in the state of automation are initiated by the user. 

adaptive automation Systems in which changes 
in the state of automation can be initiated by either 
the user or the system. 

adverse event Any undesirable outcome in the 
course of medical care. An adverse event need not 
imply an error or poor care. 

appraisal An assessment of internal and external 
events made by an individual, including attribu- 
tions of causality, personal relevance, and potential 
for coping with the event. 

arousal A hypothetical construct representing a 
nonspecific (general) indicator of the level of stim- 
ulation and activity within an organism. 



attention The act of restricting mental activity to 
consideration of only a small subset of the stimuli 
in the environment or a limited range of potential 
mental contents. 



attentional narrowing Increased selective atten- 
tion to specific cues in the environment. It can take 
the form of focusing on specific objects or events 
or scanning of the environment such that a wide 
spectrum of events is attended but not processed 
deeply. 

augmented cognition Systems that aim to en- 
hance user performance and cognitive capabilities 
through adaptive assistance. These systems can 
employ multiple psychophysiological sensors and 
measures to monitor a user's performance and reg- 
ulate the information presented to the user to min- 
imize stress, fatigue, and information overload (i.e., 
perceptual, attentional, and working memory bot- 
tlenecks). See also adaptive automation. 

augmented reality Setups that superimpose or 
otherwise combine real and artificial stimuli, gen- 
erally with the aim of improving human perfor- 
mance and creativity. 

automation A machine agent capable of carrying 
out functions normally performed by a human. 

avatar Digital representation of real humans in 
virtual worlds. 
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basal ganglia Central brain structures that are 
associated with motor learning, motor procedures, 
and reward. 

biomathematical model of fatigue Application 
of mathematics to the circadian and sleep homeo- 
static processes underlying waking alertness and 
cognitive performance. 

brain-based adaptive automation Systems that 
follow the neuroergonomics approach and use psy- 
chophysiological indices to trigger changes in the 
automation. 

cerebral laterality Differences in left and right 
hemisphere specialization for processing diverse 
forms of information. 

cochlear implant Electronic device implanted in 
the primary auditory organ (the cochlea) that stim- 
ulates the auditory terminals in the inner ear so as 
to generate a sense of sound and partially restore 
hearing in people that are severely deaf. The electri- 
cal stimulus is arranged spatially so as to reproduce 
the natural distribution of frequency bandser- 
tonotopyever the cochlear membrane. 

cognitive work analysis (CWA) A multilevel 
systems analysis framework to classify and analyze 
errors, and identify several potential levels for in- 
terventions. CWA comprises five layers, each ana- 
lyzing a different aspect of an application domain 
(work domain, control tasks, strategies, social-or- 
ganizational structure, and worker competencies). 
CWA has been successfully applied to aviation 
safety and provides a framework for improving 
health care safety. 

CI The first event-related potential (ERP) com- 
ponent representing the initial cortical response to 
visual stimuli, with a peak latency of about 60-100 
milliseconds after stimulus onset. Whether CI is 
modulated by attention is a controversial topic in 
cognitive neuroscience studies of visual selective 
attention. 

continuous wave instrument Instrument for 
optical imaging based on light sources that are con- 
stant in intensity rather than being modulated in 
intensity or pulsed. 

cortical plasticity See neuroplasticity. 

covert attention Attention that is shifted without 
a movement of the eyes, head, or body. 



critical incidents Key events that have poten- 
tially harmful consequences. These can be either 
near misses or essential steps in the chain of events 
that lead to harm. They may provide clues to the 
root causes of repeated disasters. 

cross-modal interaction Exchange of informa- 
tion between and mutual influence of sensory 
modalities, especially in association areas of the 
brain that respond to input from different modali- 
ties, such as the activation of visual areas in 
blind patients during tactile tasks like reading 
Braille. 

Doppler effect Change in the frequency of a 
moving sound or light source relative to a station- 
ary point. 

dwell Also called/bcation or gaze. A series of con- 
secutive fixations (pauses) of the eyes between sac- 
cadic eye movements within a single area of 
interest. Visual information is extracted from the 
scene during fixations. 

ecological interface design (EID) A systems de- 
sign framework that complements the systems 
analysis framework of a cognitive work analysis 
(CWA). EID takes information requirements iden- 
tified by CWA models and turns them into specific 
interface designs that make it easier for workers to 
understand what they need to get a job done. Pro- 
viding feedback in the interface has the potential to 
minimize errors and facilitate error detection and, 
thus, error recovery. 

EEG artifact A component of the recorded elec- 
troencephalogram (EEG) signal that arises from a 
source other than the electrical activity of the 
brain. Some EEG artifacts are of physiological ori- 
gin, including electrical potentials generated by the 
heart, muscle tension, or movements of the eyes. 
Others are from instrumental sources such as am- 
bient electrical noise from equipment or electrical 
potentials induced by movement of an electrode 
relative to the skin surface. 

electroencephalogram, electroencephalography 

(EEG) A time series of measurements of electri- 
cal potentials associated with momentary changes 
in brain electrical activity in collections of neurons 
resulting from stimulation or specific tasks. EEGs 
are usually recorded as a difference in voltage be- 
tween two electrodes placed on the scalp. 
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electromyogram (EMG) A record of electric 
currents associated with muscle contractions. 

electrooculogram (EOG) Electrical potentials 
recorded by electrodes placed at the canthi (for 
monitoring horizontal eye movements) or at, 
above, or below the eyes (for monitoring vertical 
eye movements, such as blinks). 

emotion A collection of changes in the body 
proper involving physiological modifications that 
range from changes that are hidden from an exter- 
nal observer (e.g., changes in heart rate) to changes 
that are perceptible to an external observer (e.g., 
facial expression). 

episodic memory A type of memory store for 
holding memories that have a specific spatial and 
temporal context. 

error An act of commission or omission (doing 
something wrong or failing to act) that increases 
the risk of harm. Errors can be subclassified (e.g., 
as slip or mistake or other) under different classifi- 
cation schemes (taxonomies) of error. 

error-related negativity (ERN) An event-related 
potential (ERP) component that is observed at 
about 100-150 milliseconds after the onset of erro- 
neous responses relative to correct responses. The 
ERN is not a stimulus-locked ERP but is a 
response-locked component. The amplitude of the 
ERN is related to perceived accuracy, or the extent 
to which participants realize their errors. 

ethology The study of animal behavior in natural 
settings, involving direct observations of behavior, 
including descriptive and quantitative methods for 
coding and recording behavioral events. 

event rate The rate of presentation of neutral 
nonsignal stimuli in a vigilance task in which criti- 
cal targets for detection are embedded. 

event-related optical signal (EROS) A transient 
and localized change in the optical properties of 
activated cortical tissue compared to baseline val- 
ues. EROS is thought to depend on scattering phe- 
nomena that accompany neural activity. Activation 
corresponds to increases in the phase delay light 
parameter (i.e., photons' time of flight) compared 
to baseline. 



event-related potential (ERP) The summated 
neural response to a stimulus, motor, or cognitive 
event as measured at the scalp by signal averaging 
the EEG over many such events. Consists of a se- 
ries of components with different onset and peak 
latencies and scalp distribution. 

executive functions A set of cognitive processes 
that help manage intentional human behavior in- 
cluding dividing and controlling attention, main- 
taining information in mind, developing and 
executing plans, social and moral judgment, and 
reasoning. 

eye field The region of the visual field within 
which an eye movement is required to process two 
spatially separated stimuli. The stationary field is 
the region within which no movement is needed to 
process the two stimuli. The head field is the re- 
gion within which a movement of the head or body 
is needed to process both items. 

fast Fourier transform Consists of a decomposi- 
tion of a complex waveform into its component el- 
ementary parts (e.g., sine waves). The fast Fourier 
transform algorithm reduces the number of com- 
putations needed for N points in a complex wave- 
form from 2N 2 to 2N lg N, where lg is the base-2 
logarithm. 

feeling Physiological modifications in the body 
proper during an emotion send signals toward the 
brain itself, which produce changes that are mostly 
perceptible to the individual in whom they were 
enacted, thus providing the essential ingredients 
for what is ultimately perceived as a feeling. Feel- 
ings are what the individual senses or subjectively 
experiences during an emotional reaction. 

Fitts' law A model of human psychomotor be- 
havior developed by Paul Fitts. The law defines an 
index of difficulty in reaching a target, which is re- 
lated to the logarithm of the movement distance 
from starting point to center of target and width of 
that target. 

fixation A dwell between saccadic eye move- 
ments. This is the time during which visual infor- 
mation is extracted from the scene. 
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functional electrical stimulation (FES) A tech- 
nology based on the direct delivery of electrical 
stimuli to the muscles of a paralyzed patient, so as 
to restore the ability to generate and control move- 
ments. A major challenge in this technology is the 
reproduction of the muscle activation patterns that 
occur naturally during motor activities such as gait 
or manipulation. 

functional field of view (FFOV) The region sur- 
rounding fixation from which information is gath- 
ered during the course of a single fixation. 
Sometimes referred to as the useful field of view 
(UFOV), perceptual span, visual span, or visual lobe. 

functional magnetic resonance imaging (fMRI) 

A technique for collecting images of the brain 
based on blood oxygenation levels of brain tissue 
so that activation of brain regions in response to 
specific tasks can be mapped. 

functional neuroimaging Brain imaging proce- 
dures that allow an investigator to visualize the hu- 
man brain at work while it performs a task and 
includes such techniques as functional magnetic 
resonance imaging (fMRI) and positron emission 
tomography (PET). 

general adaptation syndrome A set of physiolog- 
ical responses (e.g., heart rate, blood pressure, etc.) 
to noxious stimulation that serve as physiological 
defense against the adverse effects of such stimuli. 

heads-up display (HUD) An information dis- 
play located or projected on a transparent surface 
along a pilot or drivers line of sight to the external 
world. 

hedonomics The branch of science that facili- 
tates the pleasant or enjoyable aspects of human- 
technology interaction. 

Heinrich's triangle Relationships between differ- 
ent performance factors and safety errors can be 
represented by an imaginary triangle. This simple 
model can be applied to errors that lead to injuries 
in diverse settings, including transportation, nu- 
clear power, and health care. Visible safety errors 
(at the apex of the triangle, or "above the water- 
line") include errors resulting in fatality or serious 
injury. Submerged below the waterline (toward the 
base of the triangle) are events that are theoretically 
related to injury outcome and occur more fre- 
quently but do not lead to harm. 



hemovelocity The speed at which blood flows 
through a blood vessel. 

hippocampus A brain structure located in the 
medial temporal lobes of the brain, which is very 
important for the formation of memories of daily 
episodes. The name hippocampus means seahorse 
and refers to the shape of this brain structure. 

homeostatic sleep drive A physiological process 
that ensures one obtains the amount of sleep 
needed to provide for a stable level of daytime 
alertness. It increases with wakefulness and is re- 
duced with sleep. 

human prefrontal cortex (HPFC) The region of 
the cerebral cortex that is anterior to the motor 
cortex and most evolved in humans. The prefrontal 
cortex is involved in many higher-level executive 
functions, including planning, decision making, 
and coordination of multiple task performance. 

immersion The degree of a participant's engage- 
ment in a virtual reality (VR) experience or task. 

independent component analysis (ICA) A 
data-driven analytical technique for decomposing a 
complex waveform or time series of data. The tech- 
nique assumes the complex waveform to be a lin- 
ear mixture of independent signals, which it sorts 
into maximally independent components. 

jet lag Psychophysiological disturbance induced 
by a rapid shift across time zones resulting in a 
phase difference between the brain's circadian 
pacemaker (i.e., suprachiasmatic nucleus) and en- 
vironmental time. 

lateralized readiness potential (LRP) An event- 
related potential (ERP) that occurs several hun- 
dreds of milliseconds before a hand or other limb 
movement. The LRP is an average of two difference 
waves obtained by subtracting the readiness poten- 
tial (RP) of the ipsilateral site from that of the con- 
tralateral site, for left- and right-hand responses, 
respectively. LRP provides an estimate of the covert 
motor preparation process, for example, whether 
and when a motor response is selected. 

light absorption A type of light-matter interac- 
tion that results in the transfer of the light energy 
to the matter. It typically depends on the wave- 
length of the light and the type of substance. Sub- 
stances in bodily tissue (such as water, melanin, 
and hemoglobin) can be distinguished because 
they absorb light of different wavelengths. 
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light scattering A type of light-matter interac- 
tion that results in the random deviation of the di- 
rection of motion of the light (photons) through 
matter. Within the near-infrared (NIR) wavelength 
range (600-1000 ran), the dominant form of inter- 
action between light and tissue is scattering (diffu- 
sion) rather than absorption. Within this range, 
light can penetrate several centimeters into tissue, 
approximating a diffusion process. 

magnetoencephalography (MEG) Technique 
for measuring magnetic signals produced by elec- 
trical activity in the brain in response to stimula- 
tion or specific tasks. 

mental resources An economic or thermody- 
namic metaphor for the energetic and structural 
capacities required for perceptual and cognitive 
activity. 

mental workload The demands that a task im- 
poses on an individual's limited capacity to actively 
process information and make responses in a 
timely fashion. Optimal performance typically oc- 
curs in tasks that neither underload nor overload 
an individual's mental resources. 

microsleep A period of sleep lasting a few sec- 
onds. Microsleeps become extremely dangerous 
when they occur during situations when continual 
alertness is demanded, such as driving a motor ve- 
hicle. 

mismatch negativity (MMN) An event-related 
potential (ERP) that is a difference wave obtained 
by subtracting ERPs of standard stimuli (usually 
auditory) from those of stimuli that differ physi- 
cally from the standard (e.g., in pitch or loud- 
ness). The MMN has a peak latency at about 150 
milliseconds. The MMN is considered to be an 
index of automatic processing in the auditory 
modality. 

motion correction Estimating and correcting for 
the effect of subject motion on an fMRI data set. 

motor activity-related cortical potential (MRCP) 

The electroencephalogram-derived brain potential 
associated with voluntary movements. 

motor cortex The cortical area of the human 
brain (Brodmann's area 4) that regulates motor 
movements. 



naturalistic True to life, as in a real-life task per- 
formed in a real-world setting. The setting may be 
somewhat constrained by an experimenter com- 
pletely unstructured with the observer hidden and 
the subject totally unaware of being observed (the 
most "natural" setting). 

near-infrared spectroscopy (NIRS) Measure- 
ments of the concentration of specific substances 
in living tissue, capitalizing on the differential ab- 
sorption spectra of different substances. For exam- 
ple, it is possible to estimate both absolute and 
relative (percentage change) concentrations of oxy- 
and deoxyhemoglobin in the tissue with this ap- 
proach. 

nerve growth factor (NGF) Complex molecule, 
with three polypeptide chains, which stimulates 
and guides the growth of nerve cells. NGF has 
been used to guide the growth of nerve cells inside 
glass microelectrodes, thus establishing a stable 
physical connection between brain tissue and elec- 
tronic elements of brain-machine interfaces. 

neuroergonomics The study of brain and behav- 
ior at work together in naturalistic settings. This 
interdisciplinary area of research and practice 
merges the disciplines of neuroscience and er- 
gonomics (or human factors) in order to maximize 
the benefits of each. 

neuroplasticity The process of reorganization of 
the cortex, which indicates the ability of the brain 
to learn, adapt to new experience, and recover 
from brain injury. Also refers to a set of phenomena 
that characterize the variability in neuronal con- 
nectivity and neuronal response properties as a 
consequence of previously experienced inputs and 
activities. 

neurovascular coupling The relationship be- 
tween neuronal activity and the related hemody- 
namic activity as revealed by neuroimaging (PET 
and fMRI) measures. Analyses of fMRI and O 15 
PET data assume this relationship to be linear. 

Nl An event-related potential (ERP) negative 
component, whose peak latency, scalp distribution, 
and brain localization changes according to the lo- 
cation of recording site. Nl is sensitive to atten- 
tional modulation, with attended stimuli eliciting a 
larger Nl than unattended stimuli. 
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operant conditioning The modification of be- 
havior resulting from the behavior's own conse- 
quences (e.g., positive reinforcement of a behavior 
generally will increase that behavior). 

optical imaging methods A large class of imag- 
ing methods that exploit the properties of re- 
flectance and diffusion of light through biological 
tissue such as that found in the brain. 

overt attention Attention that is shifted via a 
movement of the eyes, head, or body. 

people tracker A device using a combination of 
sensors (such as accelerometers, global positioning 
systems, videos, and others) for making synchro- 
nous observations of human movement, physiol- 
ogy, and behavior in real- world settings. 

perclos Percent eye closure. An objective, real 
time, alertness monitoring metric based on per- 
centage of time in which slow eyelid closures oc- 
cur. 

person-environment transactions Mutual inter- 
actions between individuals and the environment 
such that environmental events impact individuals 
via appraisal mechanisms and, based on these pro- 
cesses, individuals act on or modify the environ- 
ment. 

phase shift Occurs when the peak or trough of 
the circadian rhythm has been advanced or delayed 
in time. This may result in an individual experienc- 
ing wakefulness when he or she would normally be 
sleeping. 

phosphene Circumscribed light perception in- 
duced, for example, by external stimulation of the 
retina or visual cortex. 

photons' time of flight The time it takes for 
photons emitted into tissue by a source to reach a 
detector. This parameter can only be obtained with 
time-resolved instrumentation. 

PI The first positive event-related potential 
(ERP) component with a peak latency for visual 
stimuli of 70-140 milliseconds. PI is sensitive to 
both voluntary and involuntary allocation of atten- 
tion. 

positron emission tomography (PET) A com- 
puterized radiographic technique used to examine 
the metabolic activity in various tissues (especially 
in the brain). 



presence The degree to which a person feels a 
part of, or engaged in, a virtual reality (VR) envi- 
ronment. 

primary inducers Environmental stimuli that 
evoke an innately driven or learned response, 
whether pleasant or aversive. Once they are pres- 
ent in the immediate environment, primary induc- 
ers automatically, quickly, and obligatorily elicit an 
emotional response. 

primary motor cortex Region of the cerebral 
cortex immediately anterior to the central sulcus. 
Also known as Brodmann's area 4, or Ml. It con- 
tains the largest projection from the brain to the 
motor neurons in the spinal cord via the pyramidal 
tract. Neurons in the primary motor cortex (and in 
other motor areas) associated with movements of 
the arm tend to be mostly active when the hand 
moves in a particular direction, a characteristic 
known as a tuning property. 

principal component analysis (PCA) A data- 
driven analytic technique for decomposing a com- 
plex waveform or time series of data into 
independent, orthogonal, elementary components. 
See also independent component analysis (ICA). 

proprioception The sense of self in sensory mo- 
tor behavior. Proprioception refers to the percep- 
tion of one's position and configuration in space, as 
derived from a variety of (nonvisual) sensory 
sources, such as the muscle spindles that inform 
the nervous system about the state of length of the 
muscles, the Golgi tendon organs, and "copies" of 
the motor commands that drive the muscles. Pro- 
prioception is the biological basis for feedback 
control of movements. 

psychomotor vigilance test (PVT) A test of be- 
havioral alertness that measures reaction times in a 
high-signal-load sustained-attention task that is 
largely independent of cognitive ability or learning. 

P300 A slow, positive brain potential with a peak 
latency of about 300-700 milliseconds. P300 am- 
plitude is sensitive to the probability of a task- 
defined category and to the amount of attentional 
resources allocated to the task, and the latency 
of P300 reflects the time needed for perceptual 
processing and categorization, independent of re- 
sponse selection and execution. 
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receptive field Area of representation of external 
space by a neuron. The stimulation of that area ac- 
tivates the neuron representing this area, for exam- 
ple, a particular region within the visual field. 

retinal prosthesis Device for electrical stimula- 
tion of cells in the retina to create visual percep- 
tion, implanted either under the lesioned retina 
(subretinal approach) or attached to the retinal sur- 
face (epiretinal approach). 

retinotopy Principle of organization of the vi- 
sual system architecture according to the topogra- 
phy of stimulation and neural connections on the 
retina. 

root cause analysis A process for identifying key 
factors underlying adverse events of critical inci- 
dents. Harmful events often involve a concatena- 
tion of factors (personnel, training, equipment, 
work load, procedures, communication) related to 
the system and individual. 

saccade A rapid, ballistic eye movement that 
shifts the observer's gaze from one point of interest 
to another. Saccades may be reflexive, voluntary, or 
memory driven. 

saccadic suppression A sharp decrease in visual 
sensitivity that occurs during a saccade. 

search asymmetry More rapid detection of the 
presence of a distinguishing feature while search- 
ing in an array of stimuli than the absence. 

secondary inducers Entities generated by the re- 
call of a personal or hypothetical emotional event 
(i.e., thoughts and memories about the primary in- 
ducer), which, when they are brought to working 
memory, slowly and gradually begin to elicit an 
emotional response. 

simulator adaptation syndrome Transient dis- 
comfort, often comprising visceral symptoms (such 
as nausea, sweating, sighing) triggered by exposure 
to a virtual reality (VR) environment. In extreme 
cases there can be "cybersickness" or simulator 
sickness, with vomiting. 

simultaneous vigilance task A comparative 
judgment type of vigilance task in which all the in- 
formation needed to detect critical signals is pres- 
ent in the stimuli themselves and there is no need 
to appeal to working memory in target detection. 



sleep debt The cumulative effect on performance 
or physiological state of not getting enough sleep 
over a number of days. Unlike sleep debt, a sleep 
surplus cannot be accumulated. 

sleepiness (somnolence, drowsiness) Difficulty 
in maintaining the wakeful state so that the indi- 
vidual falls asleep involuntarily if not actively kept 
alert. 

social cognition That aspect of cognition in- 
volving social behavior and including attitudes, 
stereotypes, reflective social judgment, and social 
inference. 

somatic state The collection of body-related re- 
sponses that hallmark an emotion. From the Greek 
word soma meaning body. 

source localization Identification of the neural 
sources of scalp-recorded EEG or ERP potentials. 
Based on the surface-recorded ERP data and cer- 
tain parameters of the volume conductortke head 
and intervening tissues between brain and scalp — 
the brain areas involved in specific cognitive pro- 
cesses can be obtained by using a dipole fit or 
other mathematical methods. 

spatial normalization Estimating and applying a 
transformation that takes a brain image as collected 
and maps it into a standardized "atlas" space. 

spatial resolution The resolution a particular 
brain imaging technique provides regarding the 
spatial localization of neural activity. Some imaging 
techniques such as fMRl have better spatial resolu- 
tion (<1 cm) than others such as PET and ERPs. 

stress A dynamic state within an individual aris- 
ing from interaction between the person and the 
environment that is taxing to the individual and is 
appraised as a psychological or physical threat. 

structured event complex Knowledge struc- 
tures that contain two or more events and are ex- 
tended in time from seconds to a lifetime and are 
the underlying representations for plans, proce- 
dures, scripts, stories, and similar stored knowl- 
edge. 

successive vigilance task An absolute judgment 
type of vigilance task in which current stimuli 
must be compared against information stored in 
working memory in order to detect critical signals. 
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suprachiasmatic nucleus A discrete brain re- 
gion lying within the hypothalamus and responsi- 
ble for the generation of circadian rhythms in 
physiology and behavior. 

temporal resolution The resolution a particular 
brain imaging technique provides regarding the 
temporal accuracy with which neural activity can 
be measured. Some techniques, such as ERPs, have 
better temporal resolution (<1 ms) than others, 
such as fMRI. 

time-resolved (TR) instruments Methods for 
optical imaging based on light sources varying in 
intensity over time. They allow for the estimation 
of the photons' time of flight, which cannot be ob- 
tained with continuous wave (CW) instruments. 

top-down regulation Influence of higher cogni- 
tive processes (e.g., attention) on early sensory pro- 
cessing (e.g., perception of light stimuli). 

transcranial Doppler sonography (TCD) The 

use of ultrasound to provide continuous noninva- 
sive measurement of blood flow velocities in the 
cerebral arteries under specified stimulus condi- 
tions. 

transcranial magnetic stimulation (TMS) 
Technique for inducing electrical currents and thus 
neural activation in the brain by strong magnetic 
pulses delivered through the scalp and skull to the 
brain surface. Useful for temporarily inhibiting 
brain function in a circumscribed region. 



ultrasound Sound with a frequency over 
20000 Hz. 

ultrasound window The temple area of the skull 
where ultrasound energy can easily penetrate. 

vigilance Sustained attention, the ability to focus 
attention and detect critical signals over prolonged 
periods of time. 

vigilance decrement Decline in signal detection 
over time during a vigilance task. 

virtual environment An environment in which 
virtual reality (VR) scenarios, tasks, and experi- 
ments are implemented. 

virtual reality (VR) The use of computer- 
generated stimuli and interactive devices to situate 
participants in simulated surroundings that resem- 
ble real or fantasy worlds. 

working memory The mental capacity to hold 
and manipulate information for several seconds in 
a short-term memory area in the context of cogni- 
tive activity. 

workload The level of perceptual, cognitive, or 
physical demand placed on an individual; the en- 
ergetic and cognitive capacity consumed by a task. 
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photons' time of flight, definition, 394 
physical neuroergonomics, 221-235 

human brain activity in motor control tasks, 
225-231,226/ 
eccentric and concentric muscle activities, 

225-226 
internal models and motor learning, 231 
load expectation, 233 
mechanism of muscle fatigue, 227 
motor control in human movements, 227-231 
control of extension and flexion movements, 

227 
postural adjustments and control, 229-230 
power and precision grip, 227-229 
reaching movements, 229 
task difficulty 230-231 
studies of muscle activation, 225 
human brain and the work environment, 224-225 
human brain in control of muscular performance, 
222-223, 223/ 225/ 
hierarchical model of the human motor system, 

222-223, 223/ 224/ 
human motor system, 222 
overview, 233 
physiological monitoring, future prospects, 

382-384 
physiology, in virtual reality, 257, 257/ 



428 



Subject Index 



PNS. See also neurorehabilitation robotics and neuro- 

prosthetics; peripheral nervous system 
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